Can I check a directory every night and only download new files or files that have changed?

You can use the PBL's text or XML directory indices to detect new files and directories.

Compare the md5 sums of files on the server with files in your local working copy.

You can also use a tool that supports site mirroring, such as wget, to perform mirroring based on file modification times.