the swap space works as another level of memory hierarchy, the kernel swap pages out before all RAM is used in order to make room for pages that are more likely to be requested. It is very much a benefit, especially when using SSDs.

so I don’t think that having it in swap is any faster than just from disk.

That is still is faster, because even though they have the same physical access times, the data stored in swap space is already mapped to memory pages. By not using swap you are just more likely to face lower hit rates for low memory usage and thrashing for high memory usage. The slowness of the disk won’t affect your overall performance as you’ll get higher cache hit rates.

Right now for example, roughly half of my RAM is being used for caching and all of my swap is used. If I didn’t have swap, most of that space used for caching would instead be used for those cold memory pages.