\documentclass[twoside]{article}[12pt]

\usepackage{listings}
\usepackage{textcomp}
\usepackage[utf8]{inputenc}
\usepackage[escape]{votutorial}

\lstset{style=daiquiri, language=SQL}

\title {The Cosmic Web}
\author {Hendrik Heinl, Stefania Amodeo}

%\titlepicture{title-aladin}
% \renewcommand{\today}{August 31, 2018}

\begin {document}

\maketitle

\pagebreak
\begin {abstract}



\end{abstract}

\usedsoftware{\Topcat\ V4.8, \Aladin\ V11.0}


\section{Starting out}
We will use the VO tools TOPCAT and Aladin. These client
software can be found here:

Aladin: \url{\linkAladin}

TOPCAT: \url{\linkTopcat}


\section {TOPCAT, ADQL and healpix}
This tutorial addresses advanced user of VO standards and protocol, namely familiaritz with TAP/ADQL will be required. Also befriending healpix will be of benefit. The use case is along the visualiztion of the cosmic web. 



\tutstep[Finding the data]

In this first step we will search for data which will fit to the following critieria: 
\begin{itemize}
\item The data needs to provide object in the range of redshift between 0.1 and 0.2
\item The data shall be comprehensive around the position of the Coma Cluster
\end{itemize}

% Try to search registry with "red shift" and "survey" or other keywords so that the result will be SDSS, and then search for SDSS dr16 to get on the GAVO service.

% Built ADQL query based on all objects and plot that

% Elaborate about the binning, then buil ADQL healpix query. 

% Plot the web. 

% Ask Setefania Amodeo about more cosmological science.
% --- SDSS has completeness and error issues. condition errors in rs
% --- use spec_z as rs
% --- Distance "function" to compute "absolute" diameter of field of view. 
% 
% cosin(12)=rs/hopethenus
% hypthenus= rs/cosin(12)
%
%
% sin(12)=gkath/hypothen
% sin(12)*(rs/cosin(12))=gkath


% Ask JEnny Sorce for comparison with simulation data


To find feasable data we will use the VO registry. The registry is the entry point for data discovery and nowadays enables you to easily narrow the search according to your needs. To also ensure that we have powerful data access, we will use the VO TAP protocol and ADQL. Hence, we only search tap services. 

Open Topcat and click on \menu{VO}\goto \menu{Table Access Protocol}. A new window will open and after a few seconds you will see a list of TAP services pop up. When opening the window, Topcat already queried the registry in the background to obtain the list of all registered TAP services. If you already know where to find your data (e.g. if you know that data linked from a paper is available at Vizier), you can simply select the according service. We do not know yet which service will provide us with good data, so we will have to search in the registry. 

For our usecase we will need a catalogue which covers a good bit of the sky and also offers good red shift data. 





We start this use case with searching and accessing neutrino data
that is available on VO services. Start Topcat and click on
\menu{VO}\goto \menu{Cone Search}. In the Cone Search window enter
\entry{neutrino} as \key{Keywords} and click on \menu{Find Services}.
5After a few moments you see the results coming in as a list of services
providing data related to the keyword. You see that the list provides
you with some information of what the data of the service contains. In
the description you also get information about the different data
providers.


Mark different rows and watch the \key{URL} change in the field below. This meta data is meant to give you an idea of what you can expect from the services and if they are of
interest for you.

With this little example you already get an idea of what the VO
standards and protocols are about: to standardize publishing, searching and
accessing data and making it reusable. 

% Dave intercept to talk about the registry

\begin{exercise}Of course you may not be interested in neutrino data.
Try to find the messengers that are of importance for you instead: try radio,
infrared or gamma ray, or whatever is suitable for you. Note: here the
registry is searching for SCS services only. The searches for other
service protocols (SIA, TAP, SSAP) work similar. 
\end{exercise}

\tutstep[Retrieve neutrino data through a Cone Search]

For our example, we select the service provided by km3net by marking the
service by the title \key{ANTARES 2007-2017}. We will get a bunch of
neutrino events to use in the next steps.  In \key{Cone parameters}
for \key{RA} enter \entry{170}, for \key{Dec} enter \entry{25} and and give
a \key{Radius} of \entry{30} degrees. Then click on \menu{OK}. Within
seconds the data will be loaded into Topcat and you see the table appear
in the main window of Topcat.

The protocol we used is the Simple Cone Search (SCS).

% Dave talks about SCS


The table we received from the SCS is a VO-Table, which not
only contains the actual data, but also some metadata.

So let's have a look at
it. We click on \menu{Views}\goto \menu{Column Info}. Here you can get some
information about the columns. You see that the data provides us with a
modified Julian Date of when the neutrino event was ovserved. If we
scroll a bit to the right, we see the UCD (Unified Content Descriptors).

We can
plot some of the data with \menu{Graphics}, but so far we wouldn't have
much more than positions to see. 

The table we received from the SCS is a VO-Table, which provides not
only the actual data, but also some metadata. 

\tutstep[Adding catalogue data to the neutrino candidates with TAP]

In this step we want to combine our neutrino data with catalogue data
from SDSS. Catalogue data typically is table data, so we will use the
Table Access Protocol and the corresponding query language ADQL to do
so. There is a lot more to say about TAP and ADQL than we will do here.
A comprehensive ADQL introduction is linked in the references. 

In Topcat go to \menu{VO} \goto\menu{TAP}. The TAP window will open and
after a few moments you will see a list of services in the window.
Analogously to the Cone Search window, you can use the TAP window to
search the VO-registry for data. At \key{Keywords} we will enter
\entry{SDSS} to search for services providing us with those data. Often
you will find the most recent catalogue data from dedicated services
related by the publishers of the catalogues. You may also find them on
the TAP Service of Vizier. Many Services providers try to anticipate
which data combination might be of interest for the users. So it's not
uncommon to find surveys like 2MASS or SDSS on several services. 
as for SDSS you see a few options to chose from. We will select the
\key{GAVO data center} (Note: the reasons here is due to the performance
of the tutorial: we want query results to come in fast. Real science
takes time and it is completely acceptable to run queries for several hours).
Click on \menu{Use Service} to get to the TAP window of the Service. 

Give it a few moments to load the meta data of the service into the TAP
window. On the left you see a tree of the catalogues and tables in this
service, on the right is the metadata browser on these tables. Mark any
table and see what the description of the table and the columns tell you
about the data in the table. This meta data still is part of the TAP
standard and is very helpful for data discovery and of course for the
data access using ADQL. 

In the upper left of the meta data at \key{Find:} enter \entry{sdss} to
find the table containing this survey. In this example we just want to
collect the identifier of an object, the position in ra and dec and the
colours in the u, g, r, i and z band. We will make use of the TAP
feature TAP UPLOAD, to join our local table with results from the remote
service. In the lower window at \menu{ADQL Text}
click on \menu{Examples} \goto \menu{Upload} \goto\menu{Upload Join}. You see an example
ADQL query appearing in the field:


\lstset{style=daiquiri, language=SQL}
\begin{lstlisting}
SELECT
   TOP 1000
   *
   FROM sdssdr7.sources AS db
   JOIN TAP_UPLOAD.t1 AS tc
   ON 1=CONTAINS(POINT('ICRS', db.ra, db.dec),
                 CIRCLE('ICRS', tc.ra, tc.decl, 5./3600.))
\end{lstlisting}

Now at the first glance that may look confusing, so let's go through it
step by step, customizing it to fit our science case as we go.
Each ADQL query starts with \sql{SELECT} followed by
specifications of ``what'' to select, what to do with the selected records
and how to return the results. \sql{TOP 1000}
means the first 1000 data records will be returned, which match the
whole query. 
With \sql{*} we select all columns of matching data records. Here will
will modify the query, because we are not interested in all of the
columns of SDSS. Instead of \sql{*} we will write: 
\sql{antares.*, sdss.objid, sdss.ra, sdss.dec, sdss.u, sdss.g, sdss.r, sdss.i, sdss.z}. This
means we want to keep all columns from our uploaded table (which we will call
\sql{antares}) and add the columns from the remote table (which we will call \sql{sdss}). 
The phrase \sql{FROM sdssdr7.sources AS sdss} specifies the table of the TAP service we
want to query \sql{sdssdr7.sources} and the short name we want to use for it \sql{sdss}. 
The next lines are the query conditions. In our example we use the ADQL
built in functions that defines the upload join.
The first part \sql{JOIN TAP\_UPLOAD.t1 AS antares} specifies that we want to join the data from SDSS with 
data in the table of neutrinos from Antares that we uploaded along with the query,
and the short name we want to use for it.
Then we describe the criteria for joining the data by defining a
circle around each object in our table of Antares neutrinos, and asking the database to
check if any sources in the SDSS table lies within any of these
circles. 
A little confusing may be the \sql{1=CONTAINS}. This is due to the boolean
result returned by the function \sql{CONTAINS}. A result of \sql{1}
means \verb|true| whereas \sql{0} means \verb|false|. 
\sql{POINT} expresses a point, the right ascension and the declination
in degrees. Our query takes the coordinates from the table using the columns
\sql{ra} and \sql{dec}. 
Analogously \sql{CIRCLE} expresses a cone in space, taking \sql{ra} and
\sql{decl} from our local table. The last entry defines the radius of
the circle in an angle in decimal degrees. The whole query looks like
this: 

\lstset{style=daiquiri, language=SQL}
\begin{lstlisting}
SELECT
   TOP 1000
   antares.*, sdss.objid, sdss.ra, sdss.dec, sdss.u, sdss.g, sdss.r, sdss.i, sdss.z
   FROM sdssdr7.sources AS sdss
   JOIN TAP_UPLOAD.t1 AS antares
   ON 1=CONTAINS(POINT('ICRS', sdss.ra, sdss.dec),
                 CIRCLE('ICRS', antares.ra, antares.decl, 10./3600.))
\end{lstlisting}


Click on \menu{Run Query} and the result should come in rather fast.
Again have a look at the new table. You see that we added a few columns,
but it seemed that we lost some rows. This is due to the query: the
database did only return those data records that matched our conditions.
Not matching records are discarded.

\begin{exercise}
You can control the ADQL JOIN behaviour by using the alternatives
\sql{LEFT OUTER JOIN, RIGHT OUTER JOIN} or \sql{FULL OUTER JOIN}. The
default command is \sql{INNER JOIN}. Play around with these options,
they might become usefull in future. 
\end{exercise}

With the colors of SDSS we could now analyze the neutrino sources and do
more reasonable science. But here we want to talk about the metadata for
a moment. You remember how we clicked on \menu{Examples} in the
TAP-Service window of Topcat, and Topcat completed an ADQL query for us.
Did you notice how Topcat ''knew'' which columns contained the data to
complete positions ? And Topcat did this for both tables, our local one,
and the one in the remote service. The secret here lies in the metadata
that is provided by VO-standards. 
In Topcat's main window go to \menu{Views}\goto \menu{Column Info}. In
the new opening window you will find the metadata on the columns, like a
human readable description, the units of the data, and the Unified
Content Descriptors (UCD). These are part of the ''magic'' working in the
background of the VO. 

\tutstep[Getting a Table with DataLink]

Once you have a certain level of interoperability, you can go on and add
more functionality and interactivity between clients and servers or in
between clients. One of the standards developed for this is Datalink. Some
services and clients can already consume datalinks. Luckily, the GAVO
service, Topcat and Aladin are capable of this. So at this point make
sure that Topcat and Aladin are running. Then go to the Topcat TAP
window, select the GAVO service and run the following query to obtain a
datset containing DataLinks:


\begin{lstlisting}

SELECT
   TOP 10000
   *
   FROM sdssdr7.sources AS sdss
   JOIN TAP_UPLOAD.t1 AS antares
   ON 1=CONTAINS(POINT('ICRS', sdss.ra, sdss.dec),
                 CIRCLE('ICRS', antares.RA, antares.Decl, 5./3600.))
   JOIN lsw.plates as plates
   ON 1=CONTAINS(POINT('ICRS', plates.centeralpha, plates.centerdelta),
                 CIRCLE('ICRS', antares.RA, antares.Decl, 1.))
   AND plates.accsize<200000000

\end{lstlisting}

Note that we added another \sql{JOIN ON} with a geometry defined by the
antares data, but also on the remote table lsw.plates. Here we want to
get some information about scanned plates from the ``Landessternwarte
Heidelberg´´. The snippet \sql{AND plates.accsize<200000000} is added
because some of the FITs files exceed 1 GB of data volume, which would
take quite some time to download. So for the sake of the tutorial, we
limit the filesize. Again, click on \menu{Run Query}. It will take a few
seconds until the data is available. 
Then go Topcat's main window, click on \menu{Views}\goto
\menu{Activation Actions}. A new window will open. On the left side
click on \menu{Invoke Service} and check the box next to it. See if in
the main window as \key{Action} \entry{View DataLink Table} is selected.
Now Topcat is prepared to interprete the DataLink block of the VOTable.
Click on the flash-button below to ''Invoke Action on first row''.
\tutstep[Using DataLink]

A new window will open and show you the DataLink Table. Here you find
four rows: \entry{\#auxiliary},  \entry{\#proc},  \entry{\#this}, and
\entry{\#preview}. 
If you scroll through these four entries, you notice, that  \entry{\#auxiliary}, 
\entry{\#this}, and
\entry{\#preview} contain links to fits file, and a possible \key{Action}
is \entry{View image internally}. With this, you can open a small
window in Topcat that will either show you a 1/4 scaled image, the whole
dataset, or a small cutout of the image. 
Let's use a
proper image analysis software to have a look at the images: Aladin.
Mark \entry{\#auxiliary} and in the drop down menu at the buttom of the
window selec at \key{Type:} \entry{FITS\_Image} and next to it at
\key{Action} \entry{Send FITS Image}. Then switch to Aladin and see how
the image is loaded. You can open the Topcat Data Display and select a
different row of our table, and then repeat the step of sending the
FITS-image to Aladin. Note that whenever you hover over the ''SAMP''
planes in Aladin, how it will compute the covered area of the image and
display it in the main window. If your screen is big enough, you can
easily display Topcat and Aladin on the same desktop and switch between
them. 

\begin{exercise}
For beginners: If you are interested in more interoperability, send the table we
obtained in the earlier tutorial step to Aladin via SAMP and see where
the sources of our neutrinos are in the images. We don't claim that you
actually see the neutrino sources in the images. But if you do, let us
know, because it would be multi messenger astronomy!
Select data points in Aladin and watch how they are highlighted in
Topcat. 
\end{exercise}
 
\begin{exercise}
For advanced: If you feel ready to get into the depths of the VO: save the Table of
this last tutorial step to a VO-Table and look at it in your favourite
text editor. Can you figure out which bit is defining the DataLink
functionality ? 
\end{exercise}

\begin{exercise}
For the VO-pro: You may have noticed that we did not talk about
\entry{\#proc} above. That is because it's linked to special service
side implemented features, in this case the SODA standard, which can be
used to perform a cutout of the FITS images. Play around with it. 
\end{exercise}


% BREAK

\section{PyVO and ObsTAP}

PyVO is the Python API to the VO standards. You can use it for data
discovery and data access in the same way as you would use VO clients
like Topcat or Aladin. But you can also use it embedded in your own
code, so that you may access data remotely from an automated script. We
will show with three simple (and one not so simple) scripts how you can
use PyVO in the multimessenger context. 

\tutstep[Example 1: PyVO and TAP]

Have a brief look at the few lines in example1.py. Note, that two lines
are necessary to supress warnings -- services and clients often are a
bit off, especially in regard of recent standards. They still work, but
warnings will be raised. We don't want to pretend that everything in the
VO is perfect, but within a tutorial, warnings might be confusing. If
you are curious what's going on, simply comment the according lines and
run the script.
The script performs a very simple query on the GAVO obscore service.
Have a look how the service object is built by
\sql{pyvo.dal.TAPService()}. This is the convention within PyVO:
\sql{pyvo.dal.SERVICETYPE(parameters)}. 
The actual search is performed with the service object method
\sql{search}. This will send the query to the server, and the last line
saves the resulting VOTABLE into the same folder as the script. 

\tutstep[Example 2: PyVO and SCS]

This example shows analogously the Simple Cone Search from PyVO. The
service object is built following the convention introduced in the last
tutorial step, and as you see, we again use the service provided by
km3net to obtain a list of neutrino events. Here we chose a much smaller
radius though. We want to keep the following steps fast. Note the last
lines of the script: we save the result to a local file, but we also use
SAMP to broadcast the result to Topcat. To make this work, you need to
make sure that topcat is running before you run the script. You will
find the result as a table in Topcat now. 

% Dave intercept
\tutstep[Example 3: PyVO and Obscore]

This example now is a bit closer to the real world. The script will make use
of the neutrino data we get from example 2, which we saved locally. So we take different steps from here: we load the data
from the folder example2. Note that we have a fallback exception in the
code, in case something went wrong during the last tutorial step. We
also introduce a longer TAP query. From the first part of the tutorial
you are familiar with the TAP-upload already. But here we are comparing
our local data with a special table on the TAP service: ivoa.obscore. 
A service providing this table follows the special standard Obscore,
which is dedicated to observational data. Each obscore-service provides
this table with exactly defined columns which enables the users to use
the same query repeatedly on each obscore-service. Of course there is no
garantuee that one will receive any results, but it's making searches
all across the VO possible, and pretty easy. Here we query the gavo
dc. only, and send the results to Topcat. 



\section {Acknowledgements}
The Authors thank the ESCAPE project from the HORIZON 2020 research and
innovation programm under granz n°824064 for making this contribution to the
2021 May Interop possible. 

Dave Morris works as a Research Software Engineer at the Royal Observatory,
Edinburgh (ROE)

Hendrik Heinl works as Ingénieur d'études at the Centre de Données
astronomiques de Strasbourg (CDS)


\section{IVOA Components}

The IVOA is organised in ''Groups of interest and competence'', with
some groups being in charge of the development and evolution of
standards (mainly the Working Groups), whereas other groups are focus on
the practices of data interoperability and the demands of specific
groups within the astronomical community (e.g. the Radio astronomy IG or
the Solar System IG). IGs help to identify demands, WGs work out if and how VO
standards can reply to these demands. 

\subsection*{Working Groups}
The following WGs are established in the IVOA and are assigned a
specific focus on the VO standards. Often a standard will touch the
focus of several WGs. Then a joined session of the WGs will be held at
the Interop meetings. 


\section{Standards}
\begin{description}

\item[]\href{https://www.ivoa.net/documents/DataLink/20150617/index.html}{Datalink}

\item[]\href{http://www.ivoa.net/documents/REC/ADQL/ADQL-20081030.pdf}{ADQL}

\item[]\href{https://www.ivoa.net/documents/ObsCore/}{Obscore}

%\item[]
%\href{https://www.ivoa.net/documents/ObsLocTAP/index.html}{ObsLocTAP}

\item[]
\href{https://www.ivoa.net/documents/latest/ConeSearch.html}{SCS}

\item[] \href{https://www.ivoa.net/documents/SAMP/}{SAMP}

\item[]\href{https://www.ivoa.net/documents/TAP/20190927/index.html}{TAP}

\item[]\href{https://www.ivoa.net/documents/cover/UCD-20050812.html}{UCD}

\item[]\href{http://www.ivoa.net/documents/VOTable/20130920/REC-VOTable-1.3-20130920.html}{VOTable}
 

\end{description}

\section{References}

\begin{description}

\item[]Demleitner, M.
\href{http://docs.g-vo.org/adql/html/}{ADQL-Course}

\item[]Demleitner, M., Becker, S.
\href{https://pyvo.readthedocs.io/en/latest/}{PyVO Documentation}

\item[]Fernique, P.
\href{https://aladin.u-strasbg.fr/java/AladinManualV10.pdf}{Aladin
Documentation}

\item[]Taylor, M.
\href{http://www.star.bris.ac.uk/~mbt/topcat/#docs}{TOPCAT Documentation}

\end{description}


 

\end{document}
            
            
