I used to mount my server’s home directory using sshfs to move quite a lot directory from and to my server’s machine. It was then, and this is now :
tar cf - directory_to_move | ssh user@hostname 'cd destination_dir; tar xf - .'
This command will do the same thing as scp but when paired with gzip or bzip you can add a nice compression (gzip or bzip) on the way. The directory structure and permission should be intact thanks to the benefits of tar.
This ssh thing never cease to amaze me.