Cloud Storage Tutorial
Table of Contents
This tutorial covers access via two methods: pittgoogle-client (with some direct use of the Google Cloud API), and the gsutil CLI.
Prerequisites
Complete the initial setup. In particular, be sure to:
Python
Setup
Imports
import fastavro
from google.cloud import storage
from matplotlib import pyplot as plt
import os
from pathlib import Path
import pittgoogle
Name some things
# fill in the path to the local directory to which you want to download files
local_dir = ''
my_project_id = os.getenv('GOOGLE_CLOUD_PROJECT')
pgb_project_id = pittgoogle.utils.ProjectIds.pittgoogle
Download files
Download alerts for a given objectId
objectId = 'ZTF17aaackje'
bucket_name = f'{pgb_project_id}-ztf-alert_avros'
# Create a client and request a list of files
storage_client = storage.Client(my_project_id)
bucket = storage_client.get_bucket(bucket_name)
blobs = bucket.list_blobs(prefix=objectId)
# download the files
for blob in blobs:
local_path = f'{local_dir}/{blob.name}'
blob.download_to_filename(local_path)
print(f'Downloaded {local_path}')
Plot cutouts and lightcurves
The functions in this section were adapted from https://github.com/ZwickyTransientFacility/ztf-avro-alert/blob/master/notebooks/Filtering_alerts.ipynb.
Open a file (see the previous section to download files)
paths = Path(local_dir).glob('*.avro')
for path in paths:
with open(path, 'rb') as fin:
alert_list = [r for r in fastavro.reader(fin)]
break
alert_dict = alert_list[0] # extract the single alert packet
print(alert_dict.keys())
Plot cutouts
pittgoogle.figures.plot_cutouts(alert_dict)
plt.show(block=False)
Cast to a dataframe and plot lightcurves
lc_df = pittgoogle.utils.Cast.alert_dict_to_dataframe(alert_dict)
pittgoogle.figures.plot_lightcurve(lc_df)
plt.show(block=False)
Plot everything together
pittgoogle.figures.plot_lightcurve_cutouts(alert_dict)
plt.show(block=False)
Command line
See also:
Get help
gsutil help
gsutil help cp
Download a single file
# fill in the path to the local directory to which you want to download files
local_dir=
# fill in the name of the file you want. see above for the syntax
file_name=
# file_name=ZTF17aaackje.1563161493315010012.ztf_20210413_programid1.avro
avro_bucket="${pgb_project_id}-ztf-alert_avros"
gsutil cp "gs://${avro_bucket}/${file_name}" ${local_dir}/.