It will affect the type of fragmentation you might suffer.
A smaller allocation unit size means more file system overhead in terms of the tables telling which units belong to which files. It also is more likely to lead to External Fragmentation which leads to more Data Fragmentation. External Fragmentation is basically referring to the space outside of (external to) the allocated file space. This space may be divded up into many small chunks so that if a large chunk is needed the request might either fail (typical in memory allocations for RAM, where space needs to be contiguous), or be forced to allocate many seperate pieces to fullfill the request (typical of disk allocations), which means the allocated space will be spread over a larger area. If the allocations are spread over a disk, it will increase the disk access time as the read heads will need to spend more time seeking (for large file accesses). For small files it might be faster to have a small allocation unit size because the files will be packed in closer together, so there will be less disk seeking, and less rotational delay to get to the next file. This later bit where the data (large files) is spread over a large area in many small chunks is Data Fragmentation. Some disk space allocation algorithms are designed to minimize this sort of fragmentation (I believe the BSD file system is one of them), and the fragmentation can also be reduced by periodically running some kind of defragmentation algorithm.
A larger allocation unit means there is more Internal Fragmentation (data that is allocated but not used, i.e. space inside/internal to the allocated chunks). This means more wasted space per file (on average). Whatever you set the allocation size to, is the minimum amount of disk space that a file will require, as all allocated file space must be kept a multiple of that number. So if you have a 10 byte file, and a 512 byte allocation unit, then that file takes up 512 bytes of disk space, thereby wasting 502 bytes. If you set it to 64 KB, then you'd be wasting 65526 bytes of space. (1 KB = 1024 bytes => 64 KB = 65536 bytes). The space wasted depends on both the file size and the size of the allocation unit. It might very well be the same in both cases. Say if the file size is 65526 bytes, then there is only 10 bytes wasted for either a 512 byte allocation unit, or a 64 KB allocation unit. By keeping the allocation unit large, you reduce the amount of file system overhead, and you can increase the access speed for large files. There is reduced file system overhead because you only need to keep track of one allocation unit for any file up to 64 KB in size (in this example), as opposed to 128 allocation units for a 64 KB sized file if the allocation unit size was 512 bytes. This saves space in the tables that keep track of disk space, and can also (depending on the file system) lead to faster operations, such as deleting files, and allocating space for new files. It can also lead to faster read times, as the disk only needs to seek once to read a whole 64 KB block (or maybe twice if it crosses certain boundaries, or more if there are a sufficient number of errors on the disk and blocks got moved far enough). For the 512 byte allocation size, the disk might need to seek up to 128 times, which is a rather slow operation.
Basically, if you have lots of small files on the volume, you'll want a fairly small allocation size. If you have just a handful of big files on a volume, you'll probably want a large allocation size. I typically use whatever the default is for my main partition (as there is usually quite a number of files and of a large mix of files sizes on this partition), and then I have a data partition for large files like (music or movies that typically take a few MB per file, well over the allocation size) and I might set the allocation size to the highest it will go.
