\documentclass[twoside]{article}[12pt]

% include packages; if you don't like the big default margins, 
% then set the geometry BEFORE including the votutorial-style
\usepackage[a4paper, top=2cm, bottom=2cm, left=2cm, right=2cm]{geometry}

%\usepackage{latex-votut/votutorial}
\usepackage[gavo]{latex-votut/votutorial}
%\usepackage[eurovo]{latex-votut/votutorial}
%\usepackage[ivoa]{latex-votut/votutorial}
%\usepackage[customlogo]{latex-votut/votutorial}
\usepackage[utf8]{inputenc}

% for syntax highlighting:
\usepackage{listings}
\usepackage{textcomp}

% more tool-commands:
\newcommand{\CosmoSim}{%
\link{CosmoSim}{http://www.cosmosim.org/}%
}

% image directory for screenshots, further icons:
\def\dir{images}

% overwrite some defaults:
% -- icons
\renewcommand{\icon}[1]{%
  % parameter: file name
  \raisebox{-1ex}{\includegraphics[scale=\iconscale]{\icondir/#1}}%
}

\fancyhead[LE,RO]{\leftmark}


% -- styles
\renewcommand{\menu}[1]{{\color{darkblue}  \tt $\,$#1$\,$}}
%\renewcommand{\url}[1]{ \href{#1}{#1} } --- that's the default anyway
%\newcommand{\menu}[1]{\textcolor{darkgreen}{ \sffamily{#1}}}


% -- adjust listings style for syntax highlighting
\definecolor{sqlkey}{HTML}{770088}
\definecolor{sqlnum}{HTML}{116644}
\definecolor{sqltable}{HTML}{0055aa}
\definecolor{sqlstring}{HTML}{aa1111}
\definecolor{sqlcomment}{HTML}{aa5500}

\lstdefinestyle{daiquiri}{%
   	frame=single,%
  	rulecolor=\color{sqlborder},%
   	backgroundcolor=\color{sqlbg},%
   	%aboveskip=0pt,%
   	%belowskip=0pt,%
	basewidth=0.5em,%
	lineskip=-0.01pt,%
   	showspaces=false,
   	showtabs=false,
   	upquote=true,
	basicstyle={\normalsize \tt \color{darkgrey}},%
	keywordstyle=\color{sqlkey},% for language specific keywords
	keywordstyle={[2]\color{sqltable}},% for table and column names, manually enclose with ` in LaTeX-source
	keywordstyle={[3]\color{sqlnum}},% for numbers in LaTeX-source
	stringstyle=\color{sqlstring},
	commentstyle=\color{sqlcomment},
}

\lstdefinelanguage{SQL}{%
	morekeywords={SELECT, FROM, WHERE, ORDER, BY, LIMIT, TOP, LIKE, AS, COUNT, GROUP, FLOOR, AND, OR, BETWEEN},%
	morekeywords={[3],1,2,3,4,5,6,7,8,9,0},% ... not working, coloring numbers is not supported ...
	sensitive=false,%
	breaklines=true, % wrap lines, if not fitting
	numbers=none, % no line numbers
	morecomment=[l]{--},%
	morestring=[s]{'}{'},%
	moredelim=[is][keywordstyle2]{`}{`},%
	%moredelim=*[s][\itshape]{/*}{*/}
}

\lstset{style=daiquiri, language=SQL}

% maketitle options
\title{Introduction to simulation databases\\ using CosmoSim}
\author{Kristin Riebe\thanks{Leibniz-Insitute for Astrophysics Potsdam (AIP); email: kriebe@aip.de}, GAVO}
\date{July 05, 2016}

\def\abstractname{Introduction}

\begin{document}
\maketitle

%{\color{red} \logooptions} 

\begin{abstract}
Simulation databases like the Millennium Database
%\footnote{\url{http://gavo.mpa-garching.mpg.de/Millennium/}}
or CosmoSim
%\footnote{\url{https://www.cosmosim.org}} 
contain data sets from cosmological simulations. Each user has access to
catalogues of dark matter halos and their history (merger trees), supplemented by (mock)
galaxy catalogues and information about the density field or tables with
the simulation particles themselves and other data sets. Access to these 
data is possible via a web interface directly with the
browser; for larger data it is more convenient to access the data via
\Topcat, wget and using their TAP/UWS interface, if available.
This tutorial gives a quick overview on the CosmoSim web
interface and shows an example use cases for extracting the mass function of a simulation, particles for a halo, the mass growth history and progenitors of a halo as well as a slice of the density field.\\
This document is a short version which contains parts of the simulations.pdf and simulation-teachers.pdf available at the GAVO tutorials webpage\footnote{\url{http://www.g-vo.org/pmwiki/Documents/Tutorials}}.
More demos, examples and tutorials can be found at the \CosmoSim\ webpage.
\end{abstract}

\usedsoftware{\CosmoSim\ web interface, \Topcat}

\section{The web interface}
%This section shows how to make queries with the \CosmoSim\ web interface.
\begin{enumerate}
    \item Open \url{https://www.cosmosim.org} in a web browser. This webpage provides access to simulations from different projects, with data sets like dark
matter halo catalogues and merger trees. The available simulations and data are described in the \key{Simulations} section of the documentation. 
There are also many documentation pages on the database and table structures, example queries and a demo video\footnote{\url{https://www.cosmosim.org/cms/documentation/demos-and-tutorials/first-steps-with-cosmosim/}}. 
You don't need to look at this now, just be aware that there is extensive documentation (and it may even be more up-to-date than this tutorial).

\pagebreak[2]

    \item \textbf{Query interface:}
    	\begin{itemize}

		\item Login with your username and password (or as the demo user/guest) and navigate to the \key{Query interface} (\url{https://www.cosmosim.org/query}).\\

    \begin{minipage}[c]{0.45\linewidth}   

	    %\centerline{\includegraphics[width=80mm]{\dir/Screenshot-CosmoSim-QueryForm}}

	    \item SQL queries are written directly into the main text area. The dialect used here is MySQL.
	     
	    Write the following text into the form:	    
   		\begin{lstlisting}
SELECT * FROM `MDR1`.`FOF`
WHERE snapnum=85
ORDER BY mass DESC LIMIT 100
		\end{lstlisting}
		This query will retrieve the 100 most massive clusters (ordered by
mass) for redshift 0 (= snapshot number 85) from the MDR1-simulation.


    \end{minipage}
    \hfill
    \begin{minipage}[c]{0.5\linewidth}
        \includegraphics[width=\linewidth]{\dir/Screenshot-CosmoSim-QueryForm}
    \end{minipage}

		\item Enter a new \key{name} for the result table below the SQL area. This name must be unique, if you leave the field empty, 
		the web application will generate a unique name for you. 
		
		\item Select \key{Short queue} and click on \key{Submit new SQL query}.
		
		\item Each query returns results in a (private) result table, the names of these tables are listed in the \key{Jobs list} on the left side. Click on your job to see its details.
		
		\item Once your job is finished (check-mark), click the job name again, then choose the tab \key{Results Table}. Browse the table to view the properties of the most massive halos.
		\item Above the form on the right side is the \key{Examples} link. Click on it to view the available example queries that you may also wish to try for yourself. Just double-click on one of them and the corresponding SQL query will be inserted into the SQL form.
	    
	    \item Browse the available simulations and tables by clicking at \key{Database Browser} on the left side above the query form. 

	    \item If you have long running queries, be sure to select \key{Long queue} before submitting, since the short queue times out after 30 seconds. 
	    It is also wise to restrict your query to use only the columns you are really interested in. This saves query time and disk space. 


    \end{itemize}

    \item Consult the \key{documentation} (Database Structure \goto\ Tables) or the database browser to get to know which table provides which data.

\end{enumerate}

\pagebreak[3]
% === Mass function ===
\section{Mass function for FOF groups}
In this section we plot the mass function for dark matter clusters of a simulation and learn about the \key{SAMP} interface to plot the results with \key{Topcat}.
\begin{enumerate}
    \item Get the mass function
    \begin{itemize}
 	    \item Open \url{https://www.cosmosim.org} in a web browser. Login and go to the \key{Query} form. 
        \item A mass function gives the number of objects for a selected mass bin. Let's try this for the MultiDark Run 1 (\sql{MDR1}) simulation, using the \sql{FOF} table. It contains dark matter clusters found with a Friends-of-Friends algorithm. Type the following query into the Query form:
            \begin{lstlisting}
SELECT 0.5+FLOOR(LOG10(mass)) AS logmass,
COUNT(*) AS num
FROM `MDR1`.`FOF`
WHERE snapnum=85
GROUP BY FLOOR(LOG10(mass))
ORDER BY logmass
            \end{lstlisting}
            This will sort the FOF clusters into logarithmic mass bins using the \sql{FLOOR}-function. The combination of \sql{COUNT} and \sql{GROUP BY} allows to count the number of clusters per bin.
        \item \key{Submit} the query and view the \key{Results Table} after it finished.
     
    
        \item Refine the bins by using steps of 0.25:
        \begin{lstlisting}
SELECT 0.25*(0.5+FLOOR(LOG10(mass)/0.25)) AS logmass, 
COUNT(*) AS num
FROM `MDR1`.`FOF`
WHERE snapnum=85
GROUP BY FLOOR(LOG10(mass)/0.25)
ORDER BY logmass
         \end{lstlisting}
         \key{Submit} again and check the \key{Results Table}: There are more rows now (factor 4).
    \end{itemize}
     
    \item View the result 
    \begin{itemize}
        \item \key{Start \Topcat}, e.g. using \entry{java -jar topcat-*.jar} from the command line in Linux.
        \item Go back to the browser window, choose the \key{SAMP} tab of your latest result and click \key{Register with a local SAMP Hub}. You may need to add a security exception in your browser first (see instructions on web page). Authorize the SAMP connection with "Yes" in the pop-up window.
        
        If you cannot see the SAMP-link, be sure that you are logged in. Guest users cannot use the SAMP interface. If you don't have an account or SAMP does not work, you can still download the table via the \menu{Download} tab (choose e.g. ASCII VO-table) and open the file in Topcat manually.
        \item Click \key{Send table to topcat} in the Topcat-row. This sends the result table to Topcat; the table name should appear there at \menu{Tables}.
        \item In \key{Topcat}, choose the \key{Plane Plot} \icon{topcat-planeplot}. This opens a new window. In the \menu{Position} tab, choose following axes: \menu{X}: \entry{logmass}, \menu{Y}: \entry{num}.
        \item Select the \menu{Form} tab next to \menu{Position} and \menu{Subsets} and click the button \key{Add new Line form} \icon{topcat-addnewlineform}. This connects the points. 
        \item Choose \menu{Axes} in the left side to get access to the axes-properties. At tab \menu{Coords} (that's the default) tick \menu{Y Log}. This makes the y-axis logarithmic. This is your mass function of FOF dark matter clusters for the MDR1 simulation!
     
    \end{itemize}
    

    \begin{minipage}[c]{0.6\linewidth}   

    \item Experiment with mass functions for other simulations:
         How does the MDR1-mass function compare to MDPL or Bolshoi at the same redshift? How do mass functions change with the halo catalogue (use \sql{BDMV} instead of \sql{FOF} with \sql{Mvir} for the mass) and redshift?
    Here are some tips:
    \begin{itemize}
        \item Take care to use the correct snapnum for each simulation to get the same redshift for each of them. The mapping between snapnum and redshifts is given in the \sql{Redshifts} table for each simulation. For example, following snapnums correspond to redshift \sql{zred=0}: MDR1: 85, Bolshoi: 416, MDPL2: 125.
        \item In order to plot two tables in one \menu{Plane Plot} window in Topcat, 
        click the button \menu{Add new positional plot control} \icon{topcat-addnewpositionalplotcontrol.png}. Then select a table and enter x- and y-axis.
        \item For adjusting the main color and label for each table, select a table at the left side, then choose the \menu{Subsets}-tab. Here you can enter your label text at \menu{Label} and choose the \menu{Color}.

    \end{itemize}

    \end{minipage}
    \hfill
    \begin{minipage}[c]{0.35\linewidth}
        \includegraphics[width=\linewidth]{\dir/Screenshot-topcat-massfunctions}
    \end{minipage}
   
   
\end{enumerate}

\pagebreak[3]

\newpage
% === Halo particles ===
\label{sec:particles}
\section{Particles of a dark matter halo}
For a few simulations, CosmoSim also provides the snapshots of selected timesteps. Thus it is possible to extract particles for a given position or a halo. In this section, the particles of an object from the FOF-catalogue of the MDR1-simulation will be extracted and plotted with Topcat. Please note, that we use here the term \emph{halo}, though actually, the FOF catalogue returns only groups of connected particles and makes no statement if they are truly gravitationally bound or not. Thus it would be more correct to speak of \emph{FOF groups} instead, but let's keep it simple.

\begin{itemize}
    \item Open \url{https://www.cosmosim.org} in a web browser. Login and go to the \key{Query} interface. 
    \item Enter following query:
    \begin{lstlisting}
SELECT p.x,p.y,p.z 
FROM 
  `MDR1`.`Particles85` AS p,
  (SELECT particleId from `MDR1`.`FOFParticles` 
   WHERE fofId = 85000001014) AS fp 
WHERE fp.particleId = p.particleId
    \end{lstlisting}
    This query first selects the ids of all particles for the FOF-halo 85000001014 and then uses them to extract their positions from the snapshot stored in table \sql{MDR1.Particles85}.
    
    
    \item \key{Submit} the query.
    \item When the query is finished, \key{click on the job} in the \menu{Job list} and go to the \key{Plot-tab}. Choose \entry{p.x} and \entry{p.y} as X and Y axis and click 
     \key{Create Plot}. This plots the particle positions in x-y-plane and is a nice simple feature for getting a first impression of the results.
    \item Make sure that \key{Topcat} is still running. 
    
    
    \begin{minipage}[c]{0.65\linewidth} 
    \item In the browser, switch to the \key{Results Table-tab}. Scroll down below the table. You should still be registered with SAMP -- if not, then click again \menu{Register with a local SAMP Hub}. Click \key{Send table to Topcat}. 

    \item In Topcat, select the new table, then choose \menu{Graphics} \goto \menu{Cube Plot} \icon{topcat-cubeplot}. This opens a new window for \key{3D plotting}.
    
    \item Within the \menu{Cube Plot} window of Topcat, choose following axes:
    \menu{X}: \entry{p.x}, \menu{Y}: \entry{p.y} and \menu{Z}: \entry{p.z}
    
    There are so many particles, that the true shape is somewhat obscured. Fortunately, since Topcat version 4, the dots can be colored by projected density: choose the \menu{Form}-tab (next to \menu{Positions} and \menu{Subsets}) and select at \menu{Shading} the \menu{Mode}: \entry{density}. Select a colormap to your liking as well. 
    
   
     
    \end{minipage}
    \hfill
    \begin{minipage}[c]{0.30\linewidth}
        \includegraphics[width=\linewidth]{\dir/Screenshot-topcat-particles}
    \end{minipage}


 \item You can navigate round the 3D plot using the mouse:
    drag to rotate, use the mouse wheel to zoom in and out, and right-click
    (or CTRL-click) to re-center.  More help on navigation is
    available by clicking the small question mark at the bottom
    left of the window.

  \item If you look close enough, you can even detect small lumps of higher density that could represent small subhalos. You could check if halo finders have detected them as well by extracting the positions and sizes of dark matter halos in this region from the CosmoSim database and overplotting them here (see more extensive tutorials at the CosmoSim webpage or GAVO Tutorials page).

\end{itemize}

\newpage


% === Mass growth, merger tree ===
\section{Mass growth of a halo}\label{sec:massgrowth}
Dark matter halos grow with time, because they accrete loose material ("smooth accretion") as well as other dark matter halos.
This can be detected when plotting the mass growth of a halo with time.
%A plot of the mass growth can give first hints, whether the merging history of a halo was rather quite and smooth or violent with many major mergers.


\begin{itemize}

\item Go to the \key{Query interface} at the CosmoSim webpage and enter following query:

\begin{minipage}[c]{0.66\linewidth}

\begin{lstlisting}
SELECT p.fofTreeId, p.treeSnapnum, p.mass, p.size
FROM `MDR1`.`FOFMtree` AS p, 
(SELECT fofTreeId, mainLeafId FROM `MDR1`.`FOFMtree` 
  WHERE fofId = 85000001014) AS m
WHERE p.fofTreeId BETWEEN m.fofTreeId AND m.mainLeafId
ORDER BY p.treeSnapnum
\end{lstlisting}

\item[] This query selects all direct progenitors of the given halo (so-called \emph{main branch}). Adjust the \texttt{fofId}, if you want to get the mass growth of a different halo.

\item Enter a \key{result name} and \key{Submit}. Wait until it is finished and send the table to \key{Topcat}.

\item In \key{Topcat}, choose \key{Graphics} \goto\ \key{Plane Plot} \icon{topcat-planeplot}.
\item In the \key{Position} tab, choose following axes: \menu{X}: \entry{p.treeSnapnum}, \menu{Y}: \entry{p.mass}.
	The \texttt{treeSnapnum} is the time axis here: 0 is a snapshot closer to the beginning and 39 at the end of the simulation.



    \end{minipage}
    \hfill
    \begin{minipage}[c]{0.30\linewidth}
        \includegraphics[width=\linewidth]{\dir/Screenshot-topcat-massgrowth}
    \end{minipage}

\item Switch to the \key{Form}-tab and add a new line form \icon{topcat-addnewlineform} for connecting the points. 
Steep increases in the mass from one timestep to another indicate the accretion of a lot of mass at once, i.e. merging events with other dark matter halos.


\end{itemize}
   

\section{Merger tree: progenitors of a halo}\label{sec:mergertree}
Let us now find all the halos which have merged into one dark matter halo at the end of the simulation.

\begin{itemize}


	\item Go to the \key{Query interface} at the CosmoSim webpage and enter following query (again adjust the \texttt{fofId}):

    \begin{minipage}[c]{0.66\linewidth} 

\begin{lstlisting}
SELECT p.fofTreeId, p.treeSnapnum, p.x, p.y, p.z, p.size
FROM `MDR1`.`FOFMtree` AS p, 
(SELECT fofTreeId, lastProgId FROM `MDR1`.`FOFMtree` 
  WHERE fofId = 85000001014) AS m
WHERE p.fofTreeId BETWEEN m.fofTreeId AND m.lastProgId
ORDER BY p.treeSnapnum
\end{lstlisting}
    
	\item Enter a \key{result name} and \key{Submit}. When the query is finished, send the table to \key{Topcat}.

    \item In Topcat, go to \key{Graphics} \goto\ \key{Cube Plot} \icon{topcat-cubeplot}. 
    \item Select following \key{axes}:
    \menu{X}: \entry{p.x}, \menu{Y}: \entry{p.y} and \menu{Z}: \entry{p.z}.
    
    \item Switch to the \key{Form}-tab and add a new size form \icon{topcat-addnewsizeform}. Set the \menu{Size} to \entry{p.size} for scaling the dots with the average halo radius.
    
    \item Apply more color by selecting at \menu{Shading}, the \menu{Mode}: \entry{aux} and entering \menu{Aux}: \entry{treeSnapnum}. Adjust the \menu{Opaque limit} to your liking.

    \end{minipage} 
    \hfill 
    \begin{minipage}[c]{0.3\linewidth} 
        \includegraphics[width=\linewidth]{\dir/Screenshot-topcat-mtree2}
    \end{minipage}

    
    \item Now halos from the beginning of the simulation are colored black-purple, more recent progenitors have yellow-red colors. 
    This nicely shows how the progenitor halos have merged with each other at earlier times to form the final halo (red) at the center. 


\end{itemize}








\newpage
% === Cosmic Web example ===
\section{The density field with {\sc TOPCAT}}

\begin{itemize}
	\item Go to the \key{Query interface} at the CosmoSim webpage and enter following query to extract a slice of the density from the Bolshoi simulation at the location of the most massive BDM halo::
			\begin{lstlisting}
SELECT ix,iy,iz,dens FROM `Bolshoi`.`Dens256` 
WHERE snapnum=416 AND iz = 94
ORDER BY ix, iy
			\end{lstlisting}
           
           We use here the table \sql{Dens256}, which contains a $256^3$ grid, and for
   each grid cell the density value. Since we only fix the \sql{iz}-value, we will get a $256\times256$ slice in x-y plane returned. 
	
    \item \key{Submit} the query and after it finished switch to the \menu{SAMP} tab. Make sure that \key{Topcat} is still open before clicking \key{Send table to topcat}. 
\end{itemize}


\begin{itemize}

    \begin{minipage}[c]{0.66\linewidth} 
	\item In \key{Topcat}, select your table and choose \key{Graphics} \goto\ \key{Plane Plot} \icon{topcat-planeplot}. This opens a new window. 

	\item In the \key{Position} tab, choose following axes: \menu{X}: \entry{ix}, \menu{Y}: \entry{iy}.

	\item Select the \key{Form} tab next to \menu{Position} and \menu{Subsets}. Increase the size of your window (click and drag with left mouse button) or switch to full screen mode in order to see all the possible options.

	\item At \menu{Shading} choose the \menu{Mode}: \entry{aux}; and for \menu{Aux}: \entry{dens+1}. This adds density coloring to your dots, but currently your image looks nearly black.

	\item Select the \menu{Aux Axis} from the left menu to adjust its properties. Change here the \menu{Scaling} to \entry{log} instead of linear. This will apply a logarithmic scaling for the colors. Since the logarithm is undefined for negative values, we had added +1 to \texttt{dens} for shifting the values above 0 (i.e. from overdensities to relative densities). The web like structure should be visible now. 

	\item At \menu{Aux Shader} choose a color table you like, e.g. \entry{Cold} for blue shades.

	\item Switch to the \menu{Range} tab to fix the range of your density values:\\
\menu{Minimum Aux:} \entry{0.01}, \menu{Maximum Aux:} \entry{100} 
	This will make the structures more prominent.
    \end{minipage} 
    \hfill %
	\begin{minipage}[c]{0.3\linewidth}
    \includegraphics[width=\linewidth]{\dir/Screenshot-topcat-dens256-aux}\\
	\end{minipage}


	\item Inspect the same slice region at different redshifts, e.g. for snapnum 243 (z=1), 132 (z=2.6) or 36 (z=10.9). Just replace the snapnum in the query above, submit the query and view the results with Topcat. Can you see how the large scale structures formed from small overdensities in the much more homogenous early universe?

\end{itemize}


\end{document}
