Sorry I can't answer your question about non-OEM chargers as I don't own one. $39
is a lot to pay for the Apple one, I'm glad my boyfriend's 4th Gen iPod came with one (FireWire though, not USB).
It never really occured to me that you could stick a shuffle in the OEM USB charger, but I guess makes sense, and it does work according to this:
iPod: About the Apple iPod Power Adapter
Naturally Apple tells you to only buy it's own branded stuff, but what manufacturer would tell you different?