Cloud Storage Tutorial

Table of Contents

This tutorial covers access via two methods: pittgoogle-client (with some direct use of the Google Cloud API), and the gsutil CLI.

Python

Setup

Imports

import fastavro
from google.cloud import storage
from matplotlib import pyplot as plt
import os
from pathlib import Path
import pittgoogle

Name some things

# fill in the path to the local directory to which you want to download files
local_dir = ''

my_project_id = os.getenv('GOOGLE_CLOUD_PROJECT')
pgb_project_id = pittgoogle.utils.ProjectIds.pittgoogle

Download files

Download alerts for a given objectId

objectId = 'ZTF17aaackje'
bucket_name = f'{pgb_project_id}-ztf-alert_avros'

# Create a client and request a list of files
storage_client = storage.Client(my_project_id)
bucket = storage_client.get_bucket(bucket_name)
blobs = bucket.list_blobs(prefix=objectId)

# download the files
for blob in blobs:
    local_path = f'{local_dir}/{blob.name}'
    blob.download_to_filename(local_path)
    print(f'Downloaded {local_path}')

Plot cutouts and lightcurves

The functions in this section were adapted from https://github.com/ZwickyTransientFacility/ztf-avro-alert/blob/master/notebooks/Filtering_alerts.ipynb.

Open a file (see the previous section to download files)

paths = Path(local_dir).glob('*.avro')
for path in paths:
    with open(path, 'rb') as fin:
        alert_list = [r for r in fastavro.reader(fin)]
    break
alert_dict = alert_list[0]  # extract the single alert packet

print(alert_dict.keys())

Plot cutouts

pittgoogle.figures.plot_cutouts(alert_dict)
plt.show(block=False)

Cast to a dataframe and plot lightcurves

lc_df = pittgoogle.utils.Cast.alert_dict_to_dataframe(alert_dict)
pittgoogle.figures.plot_lightcurve(lc_df)
plt.show(block=False)

Plot everything together

pittgoogle.figures.plot_lightcurve_cutouts(alert_dict)
plt.show(block=False)

Command line

See also:

Get help

gsutil help
gsutil help cp

Download a single file

# fill in the path to the local directory to which you want to download files
local_dir=
# fill in the name of the file you want. see above for the syntax
file_name=
# file_name=ZTF17aaackje.1563161493315010012.ztf_20210413_programid1.avro
avro_bucket="${pgb_project_id}-ztf-alert_avros"

gsutil cp "gs://${avro_bucket}/${file_name}" ${local_dir}/.