\documentclass{article}[12pt]
\def\Angstrom{{\rm\AA}}
\usepackage{listings}
\usepackage{sidecap}

\lstset{flexiblecolumns=true,showstringspaces=False}
\lstloadlanguages{python,SQL}

\usepackage{votutorial}

\title{Outlier Analysis in Low-Resolution Spectra: DFBS and Beyond}
\author{Demleitner, M.; Mickaelian, A.; 
Knyazyan, A.;\\ Baghdasaryan, D.; Mikayelyan, G.}
\titlepicture{../res/aladin-comparison.jpg}

\begin{document}
\maketitle

\begin{abstract}
Low-resolution spectra like those resulting from objective prism
observations or the RP/BP instrument on board of the Gaia astrometry satellite
enable a wealth of interesting science.  This use case investigates one
use leading up to combining many VO resources: The identification of
misclassified objects from reference databases.
\end{abstract}

\usedsoftware{\Topcat, astropy, \Aladin, \SPLAT}

\section{Introduction}

Objective prism low-dispersion spectra are useful for search and selection
purposes, as they permit a preliminary understanding on the the nature of a
large number of objects and are sufficient to distinguish objects with definite
features.  Historically, such spectra resulted from observations using
objective prisms on Schmidt telescopes, which let astronomers
observe large parts of the sky and obtain spectra of millions
of objects.  A modern source of such low-dispersion spectra will be
ESA's Gaia satellite, which will produce more than a billion of them.
It is expected that they will become publicly available in the early
2020s.

This use case will illustrate the use of such spectra to look for
``outliers'' in catalogues of special objects -- for simplicity, we use
carbon stars here.  For more background information, in particular on
our main data collection, the Digitised First Byurakan Survey (DFBS),
see Appendix~\ref{app:background}.


\section{Obtaining a Target List}
\label{sect:targets}

There are many ways to obtain lists of objects of interest; in
particular, you might choose from any object catalog you can find in
TOPCAT's \key{VO} \goto \key{Cone Search} window.  As long as you're
working with DFBS, make sure sufficiently many objects of what you're
looking at are available off of the galactic plane (other surveys of
course have other criteria).  Also, the objects should not have
redshifts significantly impacting spectra at a resolution of several nanometers in the
optical, because we assume here that spectra are different because of
intrinsic (rather than kinematic) properties.


\begin{SCfigure}
\includegraphics[width=0.6\textwidth]{../res/topcat-gcc.png}
\caption{Discovering a candidate catalogue in TOPCAT.}
\label{fig:sourcediscovery}
\end{SCfigure}


Since DFBS has epochs in the 60ies and 70ies, if you have objects that
may have high proper motions, you should also fetch and apply the proper
motions\footnote{A tutorial on how to do this with VO tools is available
at \url{http://www.g-vo.org/tutorials/add-pms.pdf}.}.

For this tutorial, a useful source is the General Catalogue of Carbon
Stars -- there are numerous carbon stars outside of the Galactic plane,
and they're spectrally fairly distinctive thanks to the Swan bands
originating from glowing C$_2$.

To obtain that catalogue, in TOPCAT go to \key{VO} \goto \key{Cone Search}.
Enter ``General catalog carbon'' into \key{Keywords}.  Use the ``fetch
all'' trick of using 0, 0 for RA and Dec and 180 for the search radius
and double click the found service (the line with ``III/227'' in
Fig.~\ref{fig:sourcediscovery}).  See Appendix~\ref{app:simbad} for how
to obtain your source lists from Simbad.


\section{Adding Spectra}
\label{sect:addspec}

We now want to add the low-resolution spectra.  Since the spectra are so small,
they can conveniently be kept right in the database.  This means that
with a single TAP query, we can pull lots of spectra, which makes bulk
analysis somewhat easier.

\tutstep[Locate a TAP server holding DFBS spectra.]  Go to the
\key{VO} \goto \key{Table Access Protocol (TAP)} dialog
in TOPCAT and type \verb|dfbs| in the
\key{Keywords:} input.  Hit \key{Find Services}, and in the TAP service
tree, select a service offering the \verb|dfbsspec.spectra|
table\footnote{One of these should be the ArVO Byu TAP service at
\url{http://arvo-registry.sci.am/tap}.} and click \key{Use Service}.

\tutstep[Locate the table we want to query.]  In the table browser
(that's the upper left pane in the TAP window), look for
dfbs again.  You'll see a \verb|dfbsspec.spectra| table -- that's the
one we want.  Select it and have a look
at the \key{Columns} tab.  You'll see that apart from quite a bit of
information on the object observed there is a
\verb|flux| column.  You could also retrieve the \verb|spectral| column,
which contains the wavelengths belonging to each point in spectrum, but
since (on this particular dataset) they're all the same, we don't actually need
them here.

\tutstep[Create the crossmatch query.] Writing a crossmatch query may
seem a bit daunting at first, but TOPCAT helps you: With the table from
section~\ref{sect:targets} selected in the main window and our
\verb|dfbsspec.spectra| in the TAP window, hit \key{Examples} \goto
\key{Upload} \goto \key{Upload join}.  You'll get a sample query that
you will have to adorn a bit.  First, remove the \verb|TOP 1000| -- we
want as many matches as there are.

Second, it's usually a good idea to think first which columns you
actually want in order to avoid wasting storage and processing -- this
becomes more important as you start dealing with millions of rows, but
it's a good idea to develop a routine.
TOPCAT's columns browser helps doing this.  For our problem, 
we'd suggest to retrieve the columns
\verb|accref| (trust us on this one for now), \verb|specid| (so we know
what object we talk about later), \verb|ra|, \verb|dec| (we may want to
inspect the piece of sky this comes from), \verb|flux| (obviously),
\verb|magb| and \verb|magr| (photometry is always good), \verb|emulsion|
(which might be useful for debugging), \verb|cutout_link| (so we can
look at what the spectra look like on the objective prism plates),
and \verb|recno| and \verb|Names|
from our source list so we can go back there if we think we need to.  
Just replace the
\verb|*| in the query with the names of these columns.  
This would result in a query somewhat
like:

\begin{verbatim}
SELECT
   accref, specid, ra, dec, flux, 
   magb, magr, cutout_link, recno, "Names"
   FROM dfbsspec.spectra AS db
   JOIN TAP_UPLOAD.t1 AS tc
   ON 1=CONTAINS(POINT(db.ra, db.dec),
                 CIRCLE(tc."_RAJ2000", tc."_DEJ2000", 5./3600.))
\end{verbatim}

(adjust the ``t1'' to match the number of the carbon star list in 
TOPCAT's Table List; this will automatically be right if you used 
Examples to generate the query).  In case
you're wondering: We need the double quotes around \verb|Names| because that's
a reserved word in SQL; if you're choosing column names yourself, avoid
these, but since we're bound to the column names the GCCS authors chose,
we have to use this little workaround.

Run the query.  Depending on what data you're trying to upload and the
server configuration, this may fail with an error like ``Error writing
request body to server''.  If that happens to you, just switch
\key{Mode} in TOPCAT's TAP dialog to ``Asynchronous'' -- what's happened
is that you've hit the limit for upload sizes on the service in
question, and some services are stingy for synchronous (immediate)
queries but generous for async (which can be queued).

\tutstep[Limit the magnitude range] 
Since photographic emulsions had a rather severely nonlinear
response, you want to constrain the magnitude of what you're looking at
at a time, or the spectra will look fairly differently just because of
that nonlinearity.  You can make a histogram plot of \verb|magr| 
to help you decide what magnitude bracket to use,
but our adivce is: use a magnitude range from 11.5 to 13.

To do that, in TOPCAT say \key{Views} \goto \key{Row Subsets}, and in
that window, say \key{Subset} \goto \key{New subset}.  In \key{Subset
Name}, type ``sample'', and in \key{Expression} say 
\verb|magr>11.5 && magr<13|.  
If you have plotted the histogram, you'll see this subset marked.

To make this subset the one that TOPCAT works with, in TOPCAT's main
window, for \key{Row Subset} select the \textit{sample} entry.

Rather than subsetting, you'd normally use server-side selection, i.e.,
just add a \verb|WHERE magr BETWEEN 11.5 AND 13| to the query.  In this
particular case, this will make the query a lot slower for somewhat
obscure reasons.  See Appendix~\ref{app:with} to see how to write such a
query nevertheless.  There, you will also find other hints on speeding
up large upload joins (which are optional here).

\tutstep[Save the data for further processing.]  Once the query is done,
in TOPCAT's main window select \key{File} \goto \key{Save
Table(s)/Session}.  In the dialog, set \key{Location} to
\verb|specs.vot|.  That name is hard-coded in the little script we use
in the next section, so make sure you get it right.


\section{Scoring}
\label{sect:scoring}

\resettutstep

We now want to see which of the spectra deviate a lot from what we
consider ``typical'' for the object class we've selected.  To do that,
we need to compute the mean of what we've pulled and then compute the
distances (in some sense of that word) of the individual spectra from
that mean.  The spectra that are far away are the ones that probably
are misclassified.

Since TOPCAT's array facilities are not quite sufficient to do that at
the time of writing, we
drop into astropy to do that.

\tutstep[Obtain and understand the source code.]  The source for our
spectrum difference computing tool is attached to this PDF.  You can
also pull it from the document
source\footnote{\url{http://svn.ari.uni-heidelberg.de/svn/edu/trunk/arvo_dfbs/res/specdistance.py}}.
Wherever you get the file from, put it next to the \verb|specs.vot| file
you created in the last section.

Here's the relevant piece of code: 
\begin{lstlisting}[language=python]
    min_len = min(len(x) for x in spec_table["flux"])
    normalized = [normalize(flux)[:min_len] for flux in spec_table["flux"]]
    template = normalize(sum(normalized))

    diff_column = table.Column(
        data=[np.mean((n-template)**2) for n in normalized],
        name="dist_from_tpl",
        description="Distance of this spectrum from the mean"
            " spectrum of this table",
        meta={"ucd": "stat.fit.goodness"})

    score_table = table.Table([spec_table["spec_id"], diff_column])
\end{lstlisting}

So, we crop all spectra to a common length (the minimal one, as we can't
fantasize values that aren't there, but we can crop away values that
are), then normalize them to integral one, compute a template as the
mean of the result, and then make a new column (with description and UCD
-- if you take that and nothing else home from this tutorial, that's
already a win) with a difference measure between each spectrum and the
template.

\begin{figure}[thb]
\includegraphics[width=0.8\textwidth]{topcat-sortdown.png}
\caption{A histogram clearly shows that one of our spectra is quite
different from the others.  Sorting by score is one way to identify
which one is bad.}
\label{fig:histo}
\end{figure}

\tutstep[Add the scores to our table in TOPCAT.]  We amend (rather than
replace) the previous table.  In this case, that's because there is
extra information in the table not preserved by astropy; in general it's
much more flexible to aggregate tables through the sort of joins we're
doing here.

So, load the files scores.vot you've just been creating into TOPCAT.

Select \key{Joins} \goto \key{Pair Match}.  As \key{Algorithm} use ``exact
value'', then select the spectrum table (its name will probably contain
``dfbsspec.spectra'') and the scores table (which will be called
``scores.vot'' by TOPCAT), and in each select specid as column. After
you've run the match, you'll have a new table called something like
``match(foo,bar)'' (or the like).  Continue working with this table.

\tutstep[Find out the outliers.]  In a table display of your match
table, right click on the \verb|dist_from_tpl| column and choose
\key{Sort down}.  The worst-fitting spectra end up on top.  Also, plot a
histogram of the \verb|dist_from_tpl| column; you'll see there's one object
that's apart (cf. Fig.~\ref{fig:histo}).


\begin{SCfigure}
\includegraphics[width=0.6\textwidth]{../res/topcat-outliers.png}
\caption{Plotting a spectrum through a TOPCAT activation action.  By
default, you'll not see lines but dots in the plot; to fix that, go to
the \key{Form} tab in the plot window and add a \key{Line} form.}
\label{fig:outliers}
\end{SCfigure}


\section{Investigating the Outliers}

\resettutstep

Let's now see what that outlier is about.  We will make ample use of
activation actions in this; essentially, these are things TOPCAT
executes when you click on a row or a point in a plot.  Activation
actions have become
a lot more fun since TOPCAT 4.6.  Make sure you have that
(or a newer) version before proceeding.

To configure activation actions, go to \key{Views} \goto \key{Activation
Actions}.

\tutstep[Plot the spectra]  The obvious first step is to plot the
spectra themselves.  That's the ``Plot Table'' action.  Add a check mark
on it.  To see what happens, hit the flash button next to \key{Invoke now
on row 1} (or whatever row you have selected).  You should see something
like Fig.~\ref{fig:outliers}.  When you now click on some other table
row, you should see a ``normal'' spectrum close to the template we've
made.  Look at a few of them and you'll see that the spikes in our
outlier spectrum are really a bit odd.

Future versions of TOPCAT might be able to plot directly from arrays,
which would let us plot our flux column rather than retrieve the
spectrum from the server -- that would be a bit more elegant in this
case.

\tutstep[View the source images] So -- is this an extraction problem or
is this, perhaps, really an object with an extremely strong emission
line?  To get to the ground of this, let's have a look at the image this
was generated from.  To do that, configure a ``Display Image''
activation action by clicking on the respective entry in \key{Actions}.
In ``Image Location'', select the \verb|cutout_link| column.  Don't
forget to actually enable the activation action by hitting its check
button.

If you run the activation actions now (either by hitting the button or
selecting an entry), you'll see the cutout of the spectrum.  Which looks
inoccuous, which is why in this particular case we suspect a technical
problem on the end of spectrum extraction.

\tutstep[Look up things in a sky atlas.]  Let's pretend we
still suspect physics.  So, let's see how the object looks like in
reality.  A great tool to do this is the interactive sky atlas Aladin.
Start it.  As a sane default, we'll want to see how things look like
in the optical, so click \key{DSS} right above the image plane to
get the venerable Palomar Sky Survey plates.  We'll want to look at
individual stars, so \key{Zoom} in to a field of view of a few arcminutes.

\begin{figure}[thb]
\includegraphics[width=\textwidth]{../res/aladin-comparison.jpg}
\caption{A comparsion of the broadband appearance of our outlier and
some tame carbon star.}
\label{fig:skyims}
\end{figure}

\tutstep[Connect Aladin and TOPCAT.]  We now want to investigate the
objects in the table in TOPCAT in broadband photometry -- or,
actually, plain images.  To make TOPCAT tell Aladin where we're looking
at, check the ``Send Sky Coordinates'' activation action (which is
properly pre-configured because TOPCAT has enough metadata to know where
the position is encoded in each row).  Run the activation action, and
you'll see a harmless-looking star.  Could this be a Carbon star?  Well,
split the Aladin display into four view (``multiview'' just below the
image) and load, in turn, 2MASS (as near infrared), WISE (as mid
infrared) and, say, Galex.  Hit \key{match} (next to the multiview
buttons) and you should see something like the left part of the
Fig.~\ref{fig:skyims}.

\tutstep[Compare with a non-outlier.] What you should have seen in the
last step is that our star is bright far into the mid-infrared.  This is
about what we'd expect for carbon stars, right?  Well, click on one of
the non-outliers, and you'll see something like what's shown on the
right part of Fig.~\ref{fig:skyims}.  Again we'd say our outlier is a
false alarm.

\section{Discovering Spectra}

\resettutstep

\begin{figure}[thb]
\includegraphics[width=0.8\textwidth]{../res/ycvn-spectrum.pdf}
\caption{A prototypical spectrum of a carbon star; this plot was made by
discovering spectra for Y CVn in Splat's SSA dialog and plotting a
result form the HyperLeda FITS Archive (reference URL 
\url{http://leda.univ-lyon1.fr}) service.}
\label{fig:ycvnspectrum}
\end{figure}

You don't have to be content with a broadband, SED-like impression of
what the star is.  You can look for actual spectra as well.  Discovering
those involves finding spectral services in the VO that have data for
the area you're looking at and then querying them.  Here's how to do
that in Aladin.

\tutstep[Get a star you are interested in into the view.] 
Hit the column that has
\verb|fbs1659-072606.51+215328.6| for \verb|spec_id| in TOPCAT and in Aladin
zoom to make your field of view a few arcminutes (other objects may have
spectra as well, but this one is a safe fall-back).

\tutstep[Restrict discovery to spectral services.] You can discover all
kinds of data in the VO.  To only look for spectra, in Aladin's
discovery tree, open the \key{Others} branch and left-click on \key{SSA
(spectrum)}.  

A dialog will pop up, and in it there is a scan button (\lower
4pt\hbox{\includegraphics[height=13pt]{../res/aladin-coverageicon.png}}) that
lets you filter your selected resources for those that have data for
your current view.  Aladin will ask you whether you're sure you want to
query more than 100 services -- just agree (unless you'd be querying
1000s of services, in which case you messed up step 1; don't worry,
these kinds of queries are fairly cheap).

When you've done that, you'll see the color of the service entries turn
green or orange depending on whether or not they have data.  Any green
one you can mouse over, check \key{in view} and hit \key{Load}.
We suggest LAMOST DR3 from org.gavo.dc at the time of writing.

\tutstep[Select a spectrum.]  After a little while, the results will be
in.  If there is a point on the object you're interested in, click on
it; if not, try a different result star.  

After clicking on a point from your spectral result plane, you should
see at least one matching spectrum in the result area below Aladin's
image display (you may need to pull it up if there are just three little
arrows instead of a result area).

To view the mid-resolution spectra, having an actual spectral analysis
program like Splat would be handy.  If you don't have it, TOPCAT will
do, too, but in that case make sure you're steering clear of image/fits
spectra, because TOPCAT can't deal with those.  

\begin{figure}[htb]
\includegraphics[width=\textwidth]{../res/withsplat.png}
\caption{The LAMOST spectrum of our fbs1659-072606.51+215328.6 example,
together with the Aladin result display for the LAMOST SSA-query and the
action we used to send the spectrum to Splat.}
\label{fig:withsplat}
\end{figure}


To load a spectrum, look for button-like objects in the result display
(cf.~Fig.~\ref{fig:withsplat}).  Don't use previews here (which for the
LAMOST DR3 service are the first button-like things).  Instead, scroll
right until you see the accref column.  Click on an accref and tell
Aladin to send your spectrum to Splat (or TOPCAT).  You can plot it
there, which yields a display like the in Fig.~\ref{fig:withsplat}.

Comparing with Fig.~\ref{fig:ycvnspectrum}, it's fairly clear that our
test star is, indeed, a Carbon star.  Of course, given that here we've
even started with one in our main bunch, that doesn't exactly come as a
surprise.

\appendix

\section{Background Information}
\label{app:background}

\subsection{More on Low Dispersion Spectra}

Even with the low resolution given by such surveys -- we
will be using the DFBS (see below) in this example, which has about
$50\,\Angstrom$ per bin --, one can distinguish many spectral lines
(absorption lines of white dwarfs and some subdwarfs, emission lines of
QSOs and other AGN, sometimes cataclysmic variables), and follow the
spectral energy distribution and colors.  The discoverabiltiy of the
lines depends on their equivalent width -- clearly, when
summarizing flux over $50\,\Angstrom$ range only very prominent features
will be discernable in the spectra.

The large coverage of collections of low-dispersion spectra also 
helps the optical identification of non-optical sources, for instance
from X-ray, IR, and radio observations.  While direct images give only the
positional difference and brightness, 
spectra give an understanding on the nature of
objects, and having our knowledge on possible counterparts of the given
source (AGN, CVs, WDs, late-type stars, etc. for X-ray sources,
galaxies, QSOs, late-type stars, etc. for IR, and mostly extragalactic
objects for radio). Even when the true counterpart is too faint and
was not observed in the spectral survey, the spectra help to exclude the detected objects as
non-relevant for the given source.

When, as in this use case, one wants to use low-dispersion spectra for
object classification,
there are a few object classes that are particularly well suited because
of their unique spectral characteristics.  For the DFBS data set,
examples include:

\begin{itemize}
\item UV-excess (Markarian) galaxies; this was the main goal of the
survey and 1515 such objects have been found (these are spectra much bluer
that those of ordinary galaxies).

\item Quasars (QSOs; blue spectra with (often) emission lines; high
redshift  QSOs are red spectra somewhat similar to
late-type stars), other AGN and peculiar galaxies (blue spectra with
(sometimes) emission lines).

\item starburst galaxies, due to the length of the blue part.

\item white dwarfs (very blue spectra with (often) absorption lines).

\item hot subdwarfs (very blue spectra with (sometimes) absorption
lines).

\item cataclysmic variables (blue spectra with (sometimes) emission
lines).

\item central stars of planetary nebulae (very blue spectra with (sometimes)
absorption lines, similar to WDs and hot sd-s).

\item planetary nebulae (emission lines).

\item late-type M stars (short spectra with very weak tail to the blue
part; absorption bands are also observed).

\item carbon stars (C; short spectra with sharp edge, very often
resembling dots for faint objects; bright objects show spectra similar
to triangles; absorption bands are also observed).
\end{itemize}


\subsection{Some Background on the DFBS}


In this use case, we will use the Digitized First Byurakan Survey (DFBS)
as the basic spectra collection.  It is the digitized version of the
famous Markarian survey, also called the First Byurakan Survey (FBS). As
of 2019, it
is still the largest spectroscopic database in the world, providing some
42,000,000 low-dispersion spectra.

The FBS has
been carried out by Markarian, Lipovetski and Stepanian in 1965-1980 on
the Byurakan Astrophysical Observatory (BAO) 102/132/213cm (40"/52"/84")
Schmidt telescope with $1.5^\circ$ prism. 2050 Kodak IIAF, IIaF, IIF,
and 103aF photographic plates in 1133 fields ($4^\circ\times4^\circ$
each, the size being $16\, {\rm cm}\times16\,{\rm cm}$) have been taken. 
The FBS covers
$17,000$ square degrees of the sky at high galactic latitudes 
($|b| > 15^\circ$); due to the telescope location, only declinations
north of $\delta=-15^\circ$ are covered.

The limiting magnitude of the survey is
$16.5^{\rm m}\cdots19.5^{\rm m}$ in V depending on the plate.  
For the majority of the plates
it is $17.5^{\rm m} \cdots 18^{\rm m}$.  The dispersion is about
$1800\,\Angstrom/{\rm mm}$ near H$\gamma$ and $2500\,\Angstrom/{\rm mm}$
near H$\beta$. The spectra obtained from the plates cover the range
$3400\cdots 6900\,\Angstrom$, and there is a sensitivity gap near
$5300\,\Angstrom$, dividing the spectra into red and blue parts. Each
FBS plate contains low-dispersion spectra of some 15,000 to 20,000
objects.

Note that Gaia's RP/BP spectra will of course reflect the technical
progress of 50 years over DFBS -- they will be much more homogeneous,
better calibrated, and there will be about a hundred times more of them.
So -- if it works just so with DFBS, it'll work marvellously with Gaia!


\section{Going Further: Confirming Doubtful Cases}
\label{app:simbad}

While we would bet that there \emph{are} misclassifications even in
respected sources like the General Catalogue we have been
working with so far, it is of course much more interesting to try and
confirm doubtful cases.  Like, for instance, things marked as doubtful
in Simbad.

So, let's repeat the exercise we just did with candidates for carbon
stars from Simbad.  We will query Simbad through TAP.  So, in TOPCAT,
go to \key{VO} \goto \key{TAP Query}.  Enter \verb|Simbad| in
\key{Keywords} and hit \key{Find Services}.  Double-click on the label
for Simbad TAP.  

As you can see when inspecting the columns for Simbad's \verb|basic|
table, there is a column otype, which has more or less magic values.  To
see what object classes Simbad might have for our problem, query the
\verb|otypesdef| table, perhaps like this:

\begin{verbatim}
SELECT * FROM otypedef
WHERE otype_longname like '%Carbon%'
\end{verbatim}

If you're new to TAP and wonder how we figured out the
\verb|otype_longname|: Select the table in TOPCAT's table browser and
check the \key{Columns} tab.  And LIKE does a wildcard query.  A bit
funkily, SQL's ``any character'' wildcard (the shell's question mark) is
the underscore, whereas ``zero or more characters'' (the asterisk in the
shell) is the percent sign.

You'll see that the class for carbon star candidates
is \verb|C*?|.  For each of these we want a position (that's what we use
to search in DFBS) and the \verb|main_id| if we ever want to refer back
to Simbad with our findings.

\begin{verbatim}
SELECT main_id, ra, dec 
FROM basic 
WHERE otype='C*?'
\end{verbatim}

With this list, repeat what we've been doing in
Sections~\ref{sect:addspec} and~\ref{sect:scoring}; you will, of course,
have to slightly modify the spectra-fetching query because we want to
keep different columns this time (essentially, \verb|main_id|) and we
need to qualify the \verb|ra| and \verb|dec| columns because that name
now is in both the remote and our local table.  This yields something
like:

\begin{SCfigure}
\includegraphics[width=0.6\textwidth]{../res/topcat-twogroups.pdf}
\caption{A plot of distance from the template spectrum over the
estimated r magnitude of the generating objects.  Do you have an
interpretation for the\dots well, two populations here?}
\label{fig:twogroups}
\end{SCfigure}

\begin{verbatim}
SELECT
   accref, spec_id, db.ra, db.dec, flux, 
   magb, magr, main_id
FROM dfbsspec.spectra AS db
JOIN TAP_UPLOAD.t1 AS tc
ON 1=CONTAINS(POINT('ICRS', db.ra, db.dec),
              CIRCLE('ICRS', tc.ra, tc.dec, 5./3600.))
\end{verbatim}

For the magnitude cut we do to cope with the nonlinear response of
photographic emulsions, let's again
create a subset; \verb|magr>11.5 && magr<13| should
do nicely again.  Don't forget to
select ``sample'' (or whatever you used as the subset name)
in \key{Row Subset} in TOPCAT's main window.

Save that, run the scoring, and join the scores as before.  Then do a
plot of \verb|magr| versus \verb|dist_to_tmpl|.  The result is fairly
suggestive (Fig.~\ref{fig:twogroups}).  If you, by the methods developed
above or any other means, manage to figure out the difference between
the ones with the small deviations and the ones up in the air, that'd be
great.  Be sure to let us know\footnote{msdemlei@ari.uni-heidelberg.de}.

\section{Server-Side Object Selection}
\label{app:with}

If you want to have server-side constraints (like the magnitude cut)
with the upload joins, you'll notice queries to which you simply add a
\verb|WHERE| run forever (actually, you'll have to
change to async mode in order to escape the timeout).  This is because
the query planner (a component of every database system) here believes
it's much smarter to first filter out all objects of, say, magnitude between
11.5 and 13, and only then use our object list, which would be much
faster.  That's something of a bug, but it's non-trivial to fix it.

To work around it,
modern TAP servers give you WITH, a SQL construct useful for many
things, but in particular for dealing with planner failures.  What's
within a WITH gets planned and executed separately.  So, re-writing the
query as:

\begin{verbatim}
WITH sample AS (
SELECT
   accref, spec_id, ra, dec, flux, 
   magb, magr, recno, "Names"
   FROM dfbsspec.spectra AS db
   JOIN TAP_UPLOAD.t1 AS tc
   ON 1=CONTAINS(POINT('ICRS', db.ra, db.dec),
        CIRCLE('ICRS', tc."_RAJ2000", tc."_DEJ2000", 5./3600.)))
SELECT * FROM sample
WHERE magr BETWEEN 11.5 and 13
\end{verbatim}

(again, adjust the ``t3'' as necessary) will make it run quickly (in a
matter of seconds).

Note that the query will also be faster if you reduce the amount of data
to upload and ingest.  For our purposes, we only need the
\verb|_RAJ2000|, \verb|_DEJ2000|, \verb|recno|, and \verb|Names| columns
from the catalog.  You'll save time and (server-side) CPU if you
restrict the upload to these columns.  To do that, hit \key{Views} \goto
\key{Column Info} in TOPCAT's main window, and in the window popping up
\key{Columns} \goto \key{Hide all Columns}.  Then in the \key{Visible}
column check the columns we need.  Incidentally, most of the actions can
also be operated from buttons, but it's harder to describe these, which
is why we're mentioning the menu entries here (as in the rest of this
tutorial).


\vfill
\onvott
\cczero
\eject
\end{document}
