Skip to content

Show how to access SPHEREx data from S3 efficiently #367

Description

@troyraen

We've encountered at least two users who say they tried to access the S3 copy of SPHEREx data but it was much slower than accessing the on-prem copy. Both seem to be cases where they tried to apply code written for local data access directly to S3 access without accounting for how to access S3 data efficiently. We have cloud data tutorials but SPHEREx data is a bit complicated so we should provides examples for it explicitly. We may also want another general cloud tutorial about common pitfalls and what to do instead.

Here's an example I found in user's code that's fine when the FITS file (hdul) is local but inefficient for S3 data.

This has to touch the data in the bucket twice (two GET requests):

if "IMAGE" not in hdul: 
    print(f"S3 FITS has no IMAGE extension")
    continue
image_hdu_orig = hdul["IMAGE"]

Change it to this so the data is only touched once:

try:
    image_hdu_orig = hdul["IMAGE"]
except ValueError:
    print(f"S3 FITS has no IMAGE extension")
    continue

I found several instances of this kind of thing in user's code. Removing the extra touches sped up the S3 code by 5-10x.

Metadata

Metadata

Assignees

No one assigned

    Labels

    content: cloudContent related issues/PRs for notebooks with cloud hosted data relevancecontent: spherexContent related issues/PRs for notebooks with SPHEREx relevanceenhancementNew notebook or feature

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions