\documentclass[twoside]{article}[12pt]

\usepackage{listings}
\usepackage{textcomp}
\usepackage[utf8]{inputenc}
\usepackage[escape]{votutorial}

\lstset{style=daiquiri, language=SQL}

\title {Astropy, PyVO and the Radio realm }
\author {Hendrik Heinl, Dave Morris}

\titlepicture{escape-logo}
% \renewcommand{\today}{August 31, 2018}

\begin {document}

\input{input_standards}

\maketitle

\pagebreak
\begin {abstract}
In the past years, VO standards were widely accepted and spread and many
astronomers get more and more familiar with standards and protocols like
Cone Search, SAMP and TAP/ADQL. Being skilled in these in combination with
Astropy and PyVO opens almost unlimited opportunities to find, to access and to
combine different datasets for further analysis.

In this tutorial the participants will learn how to use PyVO and ObsTAP
to find services, how to explore the data on these services, they will
use Datalink and SODA to perform cutouts on large images, and will use
PyVO, Matplotlib and Astropy to write a SAMP handler to combine functionality of
TOPCAT and Aladin with PyVO.

This tutorial is not an introduction in any of the used standards,
protocol tools or Python. To get more insight into this, have a look at the References section of this document.

\end{abstract}

\usedsoftware{\Topcat\ V4.8, \Aladin\ V11.0, \PyVO}

\section{Starting out}
We will use the VO tools TOPCAT and Aladin. These client
software can be found here:

Aladin: \url{\linkAladin}

TOPCAT: \url{\linkTopcat}

We will also use PyVO, Matplotlib and Astropy. These are best installed from your
distribution packages. 

A comprehensive collection of the examples in this tutorial as well as
installation scripts you will find on
\href{https://github.com/hendhd/taenaris}{github}. Checkout the taenaris
repository. 

\section{The use case}
This tutorial will show how we will show that base on a list of objects
we find photometry in SDSS, plot SEDs to find objects of interest and
eventually access radio image data from LOFAR. The premise here is that
we don't simply use uploads and x-matches, but instead will write
supporting software in Python to do a lot of the service handling for
us. A key standard for this will be the Simple Application Messaging
Protocol. The exercises of this tutorial will be in Python. To follow
them, we advice to checkout the git repository. For execises demanding
coding, we provide solutions (named in the exercise). If you struggle,
you may want to have a look into the "Helpful Links" section below. 

\section{Finding data: the VO registry}
For the first step of finding data we start Topcat. Then we load the
table objects.vot from the folde \key{pysrc} to Topcat. The content is
just a list of 19 positions to start with. For our further steps we will
make use of colours, so we are going to search for a service providing
us with data from SDSS. We will do so by using the VO-Registry from
Topcat. 

% Dave explains the VO-Registry

\REGISTRY

\tutstep[Finding services for SDSS colours]
In Topcat click on \menu{VO}\goto \menu{Table Access Protocol}. In the
window at \key{Keywords} type \entry{SDSS12} and wait for the results. Double
click on the Vizier service. The TAP Query window will open and it may
take a few seconds for the metadata to load. On the left you see a tree of the catalogues and tables on the 
service, on the right is the metadata browser on these tables. Mark any
table and see what the description of the table and the columns tell you
about the data in the table. This meta data still is part of the TAP
standard and is very helpful for data discovery and of course for the
data access using ADQL. 

In the upper left of the meta data at \key{Find:} enter \entry{SDSS12} to
find tables containing this survey. In this example we just want to
collect the identifier of an object, the position in ra and dec and the
colours in the u, g, r, and i band. We will make use of the TAP
feature TAP UPLOAD, to join our local table with results from the remote
service. In the lower window at \menu{ADQL Text}
click on \menu{Examples} \goto \menu{Upload} \goto\menu{Upload Join}. You see an example
ADQL query appearing in the field. Modify it so it looks as following:



\lstset{style=daiquiri, language=SQL}
\begin{lstlisting}
SELECT
   SDSS12, mine.*, sdss.gmag, sdss.imag, sdss.rmag, sdss.umag
   FROM "V/147/sdss12" AS sdss
   JOIN TAP_UPLOAD.t1 AS mine
   ON 1=CONTAINS(POINT('ICRS', sdss.RA_ICRS, sdss.DE_ICRS),
                 CIRCLE('ICRS', mine.ra, mine.dec, 2./3600.))
\end{lstlisting}

In case anythong goes wrong, you can use our fall back and load the file
\key{objects\_colours.vot} from the repository into Topcat to continue
the tutorial. We will need the colours in a later stage. 

\begin{exercise} 
Have a look into the metadata browser. You can find information about
the service, the catalogue, the table and the columns. This rich
metadata enables a lot of the functionality of the VO Services. 
\end{exercise}

\begin{exercise} 
You can play around with the VO registry. Search for TAP services that
provide you with data you are interested in. Try gamma ray data, AGN, or
gravitational waves and see what the VO has to offer. 
\end{exercise}

\tutstep[Example1: Finding VO services for radio images]
The good part of VO standards is: you can develop your own clients to
use them. Luckily some people did this work already: the PyVO package
provides us with an API to the VO. Here, we start
with the module \key{pyvo.registry.search}. Look into folder
\key{example1} and open the script \key{reg\_search.py}. You see the
actual code is pretty simply and self explanatory. Run the script and
look at the list of results. 

\begin{exercise} 
Modify the script in a way that it searches for TAP-services
providing gaia data (Solution: example1/exercise\_gaia.py)
\end{exercise}
\begin{exercise} 
Modify the script in a way that it searches for SIAP-services
providing HST data (Solution: example1/exercise\_HST.py)
\end{exercise}
\begin{exercise} 
Modify the script so that you search for ObsTAP services providing radio
data (Solution: example1/exercise\_radio.py)
\end{exercise}

Have a brief look at the few lines in example1.py. Note, that two lines
are necessary to supress warnings -- services and clients often are a
bit off, especially in regard of recent standards. They still work, but
warnings will be raised. We don't want to pretend that everything in the
VO is perfect, but within a tutorial, warnings might be confusing. Of
course when you are in the process of development, warnings are very
useful, so be smart with this tool of ignorance.

Remember to take a look at the helpful link section. There you will find
hints and documentation. Use a python interpreter like ipython to
recapitulate the registry query and to play with the resulting object. 

\begin{exercise} 
Try to figure out the properties of the services. In particular we are
interested in a reasonable name and a url which we can use to access the
services in a later tutorial step. (Solution: example1/exercise\_svc\_info.py)
\end{exercise}

\section{Finding data: Obscore}
The first two of above exercises are relatively easy to solve, but the
last one is a bit tricky, because though we talk about ObsTAP-services, these actually are regular tap services providing a special data model: Obscore. 

% Dave talking about Obscore

\OBSCORE

\tutstep[Example2: Making an obscore query] The big deal about obscore is: every
ObsTAP-service is providing this
exact datamodel. Thus, one can send the same query to several ObsTAP-services. Let's have a look at a obscore service: return to topcat, reopen the TAP service window,
and at \key{Keywords} enter \entry{GAVO}m click \key{Find Service} and
access the GAVO data center at ARI. Search for
\entry{ivoa.obscore}, mark the table and look and open the metadata tab
for columns. These columns are equal on all obscore services. Hence
queries written with these columns names can be performed on all obscore
service to figure out if a service provides observational at a given
position (\key{s\_ra, s\_dec}), in a specific range of time
(\key{t\_min, t\_max}) and wavelength (\key{em\_min, em\_max}). To get a
bit familiar on how to use Obscore and TAP, open
\key{example2/obscore.py}. After defining the \key{access\_url} you find
the \key{adql\_query}. This is a very basic one for now. Note that at
the bottom, we send the result of the query to Topcat. Now run the query
to get an idea what how the script behaves. 

\begin{exercise} 
Change the ADQL-query so that it is performing a Cone selection around a
given position at \entry{(ra:240, dec:47)} of 1.2 degrees. For the sake of the
tutorial keep the limit of 3 by using \key{TOP 3}(Solution:
example2/exercise\_coneselection.py)
\end{exercise}

\begin{exercise} 
Take a closer look at the metadata browser and think about alternative
solutions you could use to ask the question: ``Is my given position
within the coverage of a dataproduct on your service?'' (Solution:
example2/exercise\_region.py)
Think a moment about the difference of these two queries. 
\end{exercise}

\tutstep[Bringing Registry and Obscore together] After completing the last
exercise we now have the ADQL-query we can send to the list of services
that we found by searching the VO registry. We just have to bring these
scripts together and run all of it at once.

\begin{exercise} 
Combine the scripts. Use
example1/exercise\_svc\_info.py to iterate over the list of services,
but instead of printing the
information of each service, use the content of example2/exercise\_region.py
 and modify it to your needs. Set the limit in the ADQL-query to 15.
 Send the results of every ObsTAP-Service to Topcat via SAMP. (Solution:
 example2/exercise\_iter\_svc.py)
\end{exercise}

It's time to take a step back and look what we just did: We asked the
VO-Registry for services providing observational radio data accordingly to the
(easy access) obscore protocol, and then performed an ObsTAP-query on each of these services to ask them if they provide data at a given position. Compare
this with the length of the python script: skipping the comments, the
actual code is 13 lines (it could even be shorter, but then lose
readability). Of course the secret here lies in the VO-standards, the
metadata, and the datamodels which in combination can make data
discovery very smoothly and efficient. 

\section{Accessing data: Datalink and SODA}



\section{Interoperability: SAMP}

\section{Using Data: use UCDs for SED plotting}

\section{All together}


\section{PyVO and ObsTAP}


\SAMP

\section {Acknowledgements}
The Authors want to thank for making this contribution to ADASS 2021
possible: 

\begin{itemize}
\item ESCAPE - The European Science Cluster of Astronomy \& Particle Physics
ESFRI Research Infrastructures has received funding from the European
Union´s Horizon 2020 research and innovation programme under Grant Agreement no. 824064.

\item Markus Demleitner for sharing ideas and quite a bit of code snippets. 

\item Everybody who was involved in coming up with the SAMP standard. Well done indeed! 
\end{itemize}

Dave Morris works as a Research Software Engineer at the Royal Observatory,
Edinburgh (ROE)

Hendrik Heinl works as Ingénieur d'études at the Centre de Données
astronomiques de Strasbourg (CDS)


\section{Helpful Links}

\subsection*{Example1: Registry}

\begin{description}

\item[]\href{https://pyvo.readthedocs.io/en/latest/api/pyvo.registry.regtap.search.html#pyvo.registry.regtap.search}{pyvo.registry.search} 

\item[]\href{https://pyvo.readthedocs.io/en/latest/api/pyvo.registry.regtap.RegistryResults.html#pyvo.registry.regtap.RegistryResults}{Registry
Search Results}


\end{description}

\subsection*{Example2: Obscore}

\begin{description}

\item[]\href{http://docs.g-vo.org/adql/html/page013.html}{ADQL-Geometries}

\item[]\href{https://www.ivoa.net/documents/ObsCore/}{Obscore Standard}


\end{description}



\subsection*{SAMP}

Mtypes
SAMP on Astropy




\section{Standards}
\begin{description}

\item[]\href{http://www.ivoa.net/documents/REC/ADQL/ADQL-20081030.pdf}{ADQL}

\item[]\href{https://www.ivoa.net/documents/DataLink/}{Datalink}

\item[]\href{https://www.ivoa.net/documents/ObsCore/}{Obscore}

\item[]\href{https://wiki.ivoa.net/twiki/bin/view/IVOA/IvoaResReg}{Registry}

\item[] \href{https://www.ivoa.net/documents/SAMP/}{SAMP}

\item[]\href{https://ivoa.net/documents/SODA/20170517/}{SODA}

\item[]\href{https://www.ivoa.net/documents/cover/UCD-20050812.html}{UCD}

\item[]\href{http://www.ivoa.net/documents/VOTable/20130920/REC-VOTable-1.3-20130920.html}{VOTable}


\end{description}

\section{References}

\begin{description}

\item[]Demleitner, M. \href{http://docs.g-vo.org/adql/html/}{ADQL-Course}

\item[]Demleitner, M., Becker, S.
\href{https://pyvo.readthedocs.io/en/latest/}{PyVO Documentation}

\item[]Fernique, P.
\href{http://aladin.u-strasbg.fr/java/AladinManual6.pdf}{Aladin
Documentation}

\item[]Taylor, M.
\href{http://www.star.bris.ac.uk/~mbt/topcat/#docs}{TOPCAT Documentation}

\end{description}


\end{document}
