Stuff done on the USNO-B data in connection with the PPMX rereduction 2008
==========================================================================

Data ingestion
--------------

This mainly re-used source code from astrometry.net.

Problems:

* the star/galaxy separation flag frequently is >11 and !=19, contrary to 
  documentation. Vizier has these weird values, too, so we didn't care.
* survey=9 is mapped to survey=8 (which is what the docs imply should be done
  when you want to match the stuff up to the plate catalogue).
* aaor.table didn't have a header -- we faked one.


Validation of plate identification
----------------------------------

(Problem: It's not entirely clear how field numbers and the plate catalogue
are to be linked)

::

	select d.raj2000, d.dej2000 from usnob.data as d, usnob.plates as p where d.b1s=p.survey and d.b1f=p.field and not q3c_radial_query(d.raj2000, d.dej2000, p.alpha2000, p.delta2000, 4);

(get all entries that are offset more than four degrees from the center of
the accompanying plate on B1).

To make sure there's plate info on all objects, we ran

::

	select d.raj2000, d.dej2000 from usnob.data as d, usnob.plates as p where (d.b2s, d.b2f) not in (select survey, field from usnob.plates);

The script bin/checkPlateAssignments.py does this check.


Here's the current status:

Checking for misassigned Blue 1... Not Ok
Example for failed entry:
(221.09520555555599, -16.226377777777799, 'A', 860)
Checking for unassigned Blue 1... Ok
Checking for misassigned Blue 2... Not Ok
Example for failed entry:
(269.95030000000003, -16.268924999999999, 'E', 334)
Checking for unassigned Blue 2... Ok
Checking for misassigned Red 1... Not Ok
Example for failed entry:
(207.03614444444401, -41.472869444444399, 'F', 272)
Checking for unassigned Red 1... Ok
Checking for misassigned Red 2... Ok
Checking for unassigned Red 2... Ok
Checking for misassigned Infrared... Not Ok
Example for failed entry:
(328.06089722222202, -22.8741194444444, 'I', 601)
Checking for unassigned Infrared... Not Ok
Example for failed entry:
(97.565250000000006, 1.86879999999999, 'H', 771)


Caveats
-------

Tycho-2 stars may contribute two objects with identical raj2000 and dej2000.
Consider this bugger:

::

	select raj2000, dej2000, e_radeg, e_dedeg, epoch, pmra, pmde from usnob.data where (raj2000 BETWEEN 133.661 AND 133.662) and (dej2000 between -74.0435084 and -74.0435082);


Reconstruction of original observations
---------------------------------------

For field 250 of POSS-I Blue: 

DROP TABLE usnob.scratch;
CREATE TABLE usnob.scratch as (
    SELECT q.raj2000, q.dej2000, q.pos[0]-q.b1xi/cos(q.dej2000/180.*3.1415926536 AS alphaEp, q.pos[1]+q.b1eta AS deltaEp, q.epoch, q.b1mag FROM (SELECT a.raj2000, a.dej2000, movePm(a.raj2000, a.dej2000, a.pmra, a.pmde, '2000-01-01T00:00:00', b.epoch) AS pos,  a.b1eta, a.b1xi, b.epoch, a.b1mag FROM usnob.data AS a, usnob.plates AS b WHERE a.b1s='A' AND a.b1f=250 AND b.survey=a.b1s AND b.field=a.b1f) AS q
);
CREATE INDEX q3c_scratch ON usnob.scratch (q3c_ang2ipix(raj2000, dej2000));
CLUSTER q3c_scratch ON usnob.scratch;
ANALYZE usnob.scratch;

XXX: Do we want gnomonic projection here?


Crossmatch with PPMX
--------------------

DROP TABLE usnob.curcross;
CREATE TABLE usnob.curcross AS (
SELECT raj2000, dej2000, alphaEp, deltaEp, epoch, b1mag as mag, cmag, pos[0] as alphaPPMX,
  pos[1] as deltaPPMX, localid FROM (
   SELECT raj2000, dej2000, alphaEp, deltaEp, epoch, b1mag, cmag, movePm(alphaFloat, deltaFloat, pmra/3600., pmde/3600., '2000-01-01T00:00:00', epoch) as pos, localid from usnob.scratch as a, ppmx.data as b where q3c_join(a.raj2000, a.dej2000, b.alphaFloat, b.deltaFloat, 0.0001))
   AS q
);

CREATE INDEX q3c_curcross ON usnob.curcross (q3c_ang2ipix(alphaEp, deltaEp));
CLUSTER q3c_curcross ON usnob.curcross;
ANALYZE usnob.curcross;
GRANT SELECT ON usnob.curcross TO gavo;


A test star to see if this has worked (on A 250):

4 07 08.291	+33 17 48.58	4 07 08.225	+33 17 48.67	2435404.5	040708.3+331748	16.34	15.05



Removal of spurious entries
---------------------------

The astrometry.net people sent me a FITS containing what they identified as
spurious objects in the USNO-B catalogue.  The spurious Data descriptor
in usnob.vord takes care of pulling this into a table.

MD deleted the corresponding entries from the USNO-B table by crossmatch:

delete from usnob.data where exists (select * from usnob.spurious as a 
	where q3c_radial_query(a.ra, a.dec, raj2000, dej2000, 1/3600.))

This deleted 17327425 rows rather than 17224285, the number of objects
in that table, so we got a couple too many.  Next time, let's use 0.5/3600
as the match radius.  This is what's currently given in spurious' postCreation 
script.

The postCreation script will currently run about four days on alnilam.


An object from F42 was removed because its r1xi was NULL.


Text merge
----------

Generation of the obspos table is dreadfully slow within postgres, and 
currently we don't even need all that data in there.  Thus, we do
text dumps of that stuff.

To get a complete dump, use scripts located in usnob/bin:

dump2masscross.py
dumpppmxcross.py
makeobsposusnob.py

These leave dumps in /var/svn/space_for_gavo/dumps.  All the dumps are sorted
and can be merged together.  sort -m can handle the merge just fine
except it cannot yet read from compressed inputs.  Thus, there's
mergeDumps.sh that uses a fifo hack to retrofit that feature into sort.


Plate data reconstruction
-------------------------

Some plate data were lifted from bycd.all, a file internal to the USNO
team.  This affects I data only.  First, SERC-I.table was completely
constructed from there using bycd2table.

Second, some POSS-II plates missing in POSS-II.table were taken from there
(four of them, actually, sn0715, sn0785, sn0787, sn0859).

Third, the POSS-II plates that were actually observed with SERC-I are in
usno-b with code 7 (rather than 9, as claimed in the paper).  Data for
them (xnXXXX) was appended to POSS-II.table from another function in 
bycd.all.