I can offer two suggestions, Cobainjagstang, that have each worked for me. My slide is on me pinky so it's definitely too big, but...
First, I found that a socket wrench has a smaller hole. But the more important benefit of using a socket wrench is that it's heavier near the end of your finger, so you do not have to wiggle your forearm to get the vibrato effect...all I do is wiggle my wrist or hand and the socket wrench becomes a natural pendulum with the heavy end swinging on it's own momentum.
Second, I wind a scrap of Kleenex tissue around the end of my finger before inserting. Then I gently twist my finger into the socket to seat the tissue. If you remove your finger carefully the first few times ya play, the tissue will eventually remain inside perfectly formed to your finger.
Now there's a bit of amusement when you first set out to using a socket wrench. When you go to the hardware store to buy one, and ask the clerk, "Where are your socket wrenches, sir?", the clerk will direct you and ask, "What size do you need?". Watch the reaction when you reply, "Dunno. I'll have to try em on."