We've encountered at least two users who say they tried to access the S3 copy of SPHEREx data but it was much slower than accessing the on-prem copy. Both seem to be cases where they tried to apply code written for local data access directly to S3 access without accounting for how to access S3 data efficiently. We have cloud data tutorials but SPHEREx data is a bit complicated so we should provides examples for it explicitly. We may also want another general cloud tutorial about common pitfalls and what to do instead.
Here's an example I found in user's code that's fine when the FITS file (hdul) is local but inefficient for S3 data.
This has to touch the data in the bucket twice (two GET requests):
if "IMAGE" not in hdul:
print(f"S3 FITS has no IMAGE extension")
continue
image_hdu_orig = hdul["IMAGE"]
Change it to this so the data is only touched once:
try:
image_hdu_orig = hdul["IMAGE"]
except ValueError:
print(f"S3 FITS has no IMAGE extension")
continue
I found several instances of this kind of thing in user's code. Removing the extra touches sped up the S3 code by 5-10x.
We've encountered at least two users who say they tried to access the S3 copy of SPHEREx data but it was much slower than accessing the on-prem copy. Both seem to be cases where they tried to apply code written for local data access directly to S3 access without accounting for how to access S3 data efficiently. We have cloud data tutorials but SPHEREx data is a bit complicated so we should provides examples for it explicitly. We may also want another general cloud tutorial about common pitfalls and what to do instead.
Here's an example I found in user's code that's fine when the FITS file (
hdul) is local but inefficient for S3 data.This has to touch the data in the bucket twice (two GET requests):
Change it to this so the data is only touched once:
I found several instances of this kind of thing in user's code. Removing the extra touches sped up the S3 code by 5-10x.