Note: Of course, in each version, many minor improvements and bug fixes go in. To see the full history, use svn log http://svn.ari.uni-heidelberg.de/svn/gavo/python/trunk/ | less to see the gory details. If anything not mentioned here breaks something for you, we'd consider it a severe bug, though. Version 1.2 (2018-07-17) * New dachs start command to produce structured templates for certain service types. * Support for ADQL 2.1 draft, including almost all of the optional parts (run dachs imp //adql for some nice new UDFs). * New coverage element (with updaters) to build and declare the space-time-spectral coverage of a resource. * New odbcGrammar to pull rawdicts from remote databases. * You can now declare associated datalink services for tables (the _associatedDatalinkSvc meta item). * We now force matplotlib to read its configuration from configDir/matplotlibrc; to get a default, just run dachs init. * Now supporting VOSI 1.1; in particular, DaCHS now understands the detail hints and has per-table endpoints. * The indices generated by the ssa mixins are now a bit more sensible considering typical query modes. You probably want to dachs imp -I your ssap data collections as convenient. * ssapCore no longer wantonly adds preview columns. If you have previews with spectra, you probably want to add auto to your ssapCores. * You can now add a statisticsTarget property to columns; you will want this on largeish tables with non-uniformly distributed values to aid the query planner. * DaCHS's log now by default does not contain IP addresses, user agents, and referrers any more, which should mostly keep you from processing personal data. To get back the previous behaviour, set [web]logFormat in /etc/gavo.rc to "combined". * As usual, many minor bug fixes and improvements (e.g., memmapping FITSes for cutout again, delimited table references in ADQL, new-style tutorial resource records, correct obscore standardId) Version 1.1 (2017-12-01) * DaCHS now officially requires python 2.7. * Now supporting TAP 1.1; in particular, TOP n doesn't trump MAXREC any more, and it doesn't affect OVERFLOW indication. Also, TAP_SCHEMA is updated (this happens as a side effect of dachs upgrade). * Now serialising spoint, scircle, and friends to DALI 1.1 xtypes (timestamp, point, polygon, circle). Fields explicitly marked with adql:POINT or adql:REGION will still be serialised to STC-S. Do this only if you have no choice (DaCHS has this for obscore and epntap s_region right now). * Sanitised output column selection. This may make for slight changes in service responses, in particular in VOTable formats. See http://docs.g-vo.org/DaCHS/ref.html#output-tables * DaCHS no longer comes with an outdated version pyparsing and instead uses what's installed on the system. The Debian package further re-uses additional system resources if available (rjsmin, jquery). * DaCHS now tries a bit harder to come up with sensible names for SODA result files. * map/@source is no longer limited to identifier-like strings; any key that's in your source is fair game. * For incremental imports with data that's updated now and then, there's now ignoreSources/@fromdbUpdating. * relative imports from custom code ("import foo" in a custom core, for instance, getting res/foo.py) no longer work. See http://docs.g-vo.org/DaCHS/ref.html#importing-modules for details. * fixed a severe bug in the creation of obscore metadata from SSAP tables. If you use //obscore#publishSSAPHCD or //obscore#publishSSAPMIXC mixins, update the obscore definitions by running dachs imp -m , followed by dachs imp //obscore (only necessary once at the end). * You can now define a footer.html template that's added at the foot of the main page content. Version 1.0 (2017-07-11) * DaCHS' main entry point is now actually called dachs (i.e., call dachs imp q and such in the future). gavo will work as an alias for quite a while to come, though. * UWS support is now for version 1.1 (i.e., there's creationDate, filters in the joblist, and slow polling). * Added "declarative" licenses. Please read the Licensing chapter in the tutorial and slap licenses on your data. * Now using astropy.wcs instead of pywcs, and astropy.io.fits instead of pyfits. The respective APIs have, unfortunately, changed quite a bit. If you used them (e.g., in processors), you'll have to change your code; it's unlikely services are impacted at runtime. (see also http://docs.g-vo.org/DaCHS/howDoI.html#update-my-code). * Removed //epntap#table-2_0. Use //epntap2#table-2_0 instead (sorry). * Removed sdmCore (use SODA instead) * Removed SODA procs in //datalink (use the ones from //soda instead) * Removed imp -u flag and the corresponding updateMode parse option. If you used that or the uploadCore, just mark the dds involved with updating="True". * Massive sanitation of input parameter processing. If you've been using inputTable, inputDD, or have been doing creative things with inputKeys, please check the respective services carefully. See also "Building Service Interface" in the reference documentation. Most user-visible change: If you've been using repeated parameters to fill array-valued inputs, that's no longer allowed; if you actually must have this kind of thing, you'll need a custom core and must fill the arrays by hand. * In DaCHS' SQL interface, tuples now are matched to records and lists to arrays (was the other way round before). If while importing you manually created lists to fill to array-like columns, you'll have to make tuples from these now. * rsc.makeData or rsc.TableForDef no longer automatically make connections when used on database tables. You must give them explicit connection arguments now ("with base.getTableConn() as conn:"). * logo_tiny and logo_big are now ignored, all logos spit out by DaCHS are based on logo_medium.png, including, if not overridden, the favicon (that you will now get if you have not set it before). * Removed (probably largely unused features) editCore, SDM2 support, pkg_resource overrides, simpleView, computedCore. * Removed argparse module shipped with DaCHS. This breaks compatibility with python 2.6 (although you can still run DaCHS with a manually installed argparse.py in 2.6). * Hopefully more useful manpage -- have a peek! Version 0.9.8 (2016-12-08) * SODA/Datalink implementation is now considered stable. Go forth and make fabulous links and cutout services. Check the overhauled datalink section in the reference documentation if you have a couple of minutes: http://docs.g-vo.org/DaCHS/ref.html#datalink-and-soda * TAP_SCHEMA columns now has new column_index column. Its addition should be automatic as long as your TAP-published RDs are in order. * EPN-TAP 2.0 changed some column and parameter names, and the mixin followed. If you've already used EPN-TAP 2.0, you will have to re-import your data and slightly adapt your RD (e.g., c1_min -> c1min, t_exp_min -> time_exp_min etc). * DaCHS now emits warnings for columns whose names are SQL or ADQL reserved names (this includes popular items like area, size, and distance). Even if things have worked so far, please change these names. We guarantee you'll regret it if you don't. * Full support for VOResource 1.1 relationships (http://docs.g-vo.org/vocab-test/relationship_type). Please try them if you have appropriate data. Version 0.9.7 (2016-08-18) * Behaviour change: pythonCores no longer run in a file system sandbox (can't do that thread-safely). * Support for SIAP version 2 (see tutorial for details) * The default VOTable version delivered is now VOTable 1.3 (actually, it's been 1.3 for a while now, we just didn't declare it) * support for Obscore 1.1 (update should be automatic) * support for EPN-TAP 2.0 (manually change to the #table-2.0 mixin) * Obscore support is now at version 1.1 * DaCHS now supports CORS for web-based UIs * DaCHS's ADQL cross matches should now have "good" query plans regardless or argument order (but are half as fast as in previous. "good" cases). * SCS services publishing TAP-readable tables now get an auxiliary TAP publication automatically (you'll need to re-gavo pub them, though, if you want the Registry to notice quickly). * Removed bbox-based SIAP because DaCHS has been requiring pgsphere for quite a while now, and hence there's not need for a lousy fallback. * gavo admin xsdVal and XSD-validating regression tests now work even with package installations; so: you got your own schema mirror now! * New gavo admin updateTAPSchema command for syncing TAP_SCHEMA with the RDs. * New built-in preview-managing scheme for large data collections. * As usual, many small bug fixes and new features. Version 0.9.6 (2016-03-16) * Several new indices are now created by default; you probably want them, too. Run "gavo imp -I //products". Similarly, gavo imp -I on all RDs that have SSA tables with datalink services (if you don't know what that is, you don't have it) * New soda RD containing (almost) all to build a WD-compliant cube cutout service (and a bit more). You can update any existing services you have by changing "datalink" to "soda" in the referenced procDefs. * API change: base.makeMetaItem and makeMetaValue were a pain. They have simply gone in the expectation that nobody has actually used them. Protest if we're wrong. Also there is no type attribute on meta any more. Also, no meta/@type any more in RDs. I hope you've not really used *that*. * API change: When CondDescs yielded more than one fragement, the fragments were combined with AND so far. Now, it's OR by default (you can restore the previoius behaviour using condDesc/@joiner) * Some changes in the way the selection and evaluation of sort keys work in particular in web interfaces. If you've been playing games with this, check if what you did still works. * Support for DALI 1.1 xtype="interval" parameters. * New recommended way to specify author lists; see http://docs.g-vo.org/DaCHS/tutorial.html#authors * As usual, many small bug fixes and new features. Version 0.9.5 (2015-10-07) * **RD breakage**: The old DaCHS implemenation got the meaning of spectralSI and fluxSI completely wrong. You will fairly certainly have to change RDs doing SSA. Any occurrence of spectralSI will have to become spectralUnit, and fluxSI fluxUnit (run gavo val ALL to catch RDs where that's necessary). While you could add correct spectralSI and fluxSI values (which would look like "10e-10 L" or so), the DaCHS authors currently consider that a waste of time. * Possible service breakage: We're now using VOTable tabledata parsers to parse the values of PARAMs; the same code is used to parse HTTP parameters. If you define custom InputKeys and got metadata slightly wrong, these services might fail now. If you have InputKeys anywhere in your RDs, make sure the corresponding services still behave as expected. * Column naming is now ADQL-compliant, which means that leading underscores are no longer considered ok for regular identifiers (though they are fine to postgres). You should run a ``gavo val ALL`` and fix any warnings to the effect that something is not a "regular ADQL identifier" you may get. Also read http://docs.g-vo.org/DaCHS/commonproblems.html#column-tab-foo-is-not-a-regular-adql-identifier Since there's no harm, but it may automatically fix things, just run gavo imp -m //tap after upgrading. * SLAP support: If you have data on spectral lines, DaCHS now speaks the protocol to distribute them. * Generic UWS support: If you want to provide non-TAP UWS services (like large server-side compute jobs and such), there's now the uws.xml renderer (in combination with a pythonCore or customCore). Details at http://docs.g-vo.org/DaCHS/ref.html#custom-uwses * Sidebar style has changed; actually removed the old support code, so the old behaviour cannot be restored by templates and javascript alone, but I'd be happy to provide you with the necessary hooks. * By popular demand: The "_r" column (in cone searches, distance between the cone center and the actual object) is there. To get it, use an scsCore rather than a dbCore in your scs services. It works with the form renderer, too. * In VO services (scs.xml, siap.xml, ssap.xml), even fields from an explicit output table are now filtered by verbLevel. This means that fields you deemed unimportant might no longer show up in default answers. To get them back, either lower their verbLevel or add a query parameter VERB=3. Version 0.9.4 (2015-02-03) * Uniform referencing to RDs and elements in them, allowing to skip input-relative paths in all of gavo's subcommands. * Some fixes for Debian jessie compatibility. * fitsProdGrammar supports compressed FITSes transparently. * New templating system for writing more beautiful FITS headers in processing (see kapteyn/q in hdinputs for examples) * New grammar for working with CDF files * New api renderer for DAL-style machine-oriented services (see sp_ace/q in hdinputs for an example) * Transparent support for gzip compression on selected documents * We now officially force SIAP services to have their bandpasses in m (if this breaks your service, it's been broken before). * Possibly bookmark-relevant change: _DBOPTIONS_LIMIT is now MAXREC * Possibly scripting-relevant change: --ui option on gavo imp now needs to be given before the imp. * Possibly registry-relevant change: this upgrade will change your TAP service's IVORN (to ivo:///tap; we admit that was a dumb idea, but reverting now will only make things worse) Version 0.9.3 (2014-09-04) * Improved EPN-TAP support such that it actually works (this release is mainly for visitors of EPSC); this includes a simple interface to pyPDS. * New "api" renderer that mimics DAL parameter style (including UPLOAD and metadata VOTable) but doesn't assume particular data semantics. * The registry entry for the built-in TAP service now uses the vanity name; if you have published your TAP service to the Registry, run gavo pub //tap to have a less ugly IVORN and access URL (nothing will break if you don't). Version 0.9.2 (2014-08-14) * !Important!: Internal changes to the way mixins work cause some column sequences to change. In particular, as a consequence you must re-import all tables with non-trivial mixins. We are really sorry about that. Gavo upgrade will warn about tables it finds that need re-importing. * DaCHS can now produce JSON output (with certain not yet VO-standardised conventions) from TAP and other services. * TAP_SCHEMA type names are automatically updated to comply to recently established conventions. If you have tricked weird type names into TAP_SCHEMA (you'll know if you have), please ensure DaCHS hasn't messed it up. * DaCHS does not produce DataCollection registry records any more; they are all CatalogServices with (TAP) services exposing the data given as capabilities directly. * The generation of examples (formerly TAP examples) is now done completely differently. If you have material in __system/adqlexamples, you'll have to move it to etc/userconfig.rd (see doc on the tap renderer). * DaCHS now has a hard dependency on python-lxml (apt-get it if on Debian and not running from package, otherwise see http://lxml.de/) Version 0.9.1 (2014-04-30) * New regression testing framework derived from a local hack we've had in the GAVO DC for a while now. Try it, it's your friend. See the http://docs.g-vo.org/DaCHS/ref.html#regression-testing * New framework for having operator-defined previews. Please have a look at http://docs.g-vo.org/DaCHS/ref.html#product-previews for details. An unfortunate side effect is that all your cached previews are now invalid. Remove all files in (presumably) /var/gavo/web/previewCache. * Column selection for non-HTML now always includes all columns explicitely given in the output table (unless supressed by a displayHint). If you lie about types in your output table (e.g., declare string types for things that are really ints), you may now get errors on VOTable generation. Fix the types or have a noxml=true displayHint in these cases. * Removed the table attribute on ForeignKeys (that's been deprecated for more than a year now). Use the inTable attribute with an actual table reference instead. * Datalink support bumped to WD-2014-02-28 Version 0.9 (2013-12-13) * API CHANGE: If you have custom logic (i.e., phraseMakers) on string inputKeys with values, you'll now get back lists of strings by default. This will break your phraseMakers in that situation. Solution: Add multiplicity="single" to the inputKeys. * Change to latest VOUnit version. Many unit strings that previously appeared ok no longer are; in particular, exponentiation is now ** exclusively and no longer ^. You'll get warnings with gavo val. * Added support for Datalink for both cutout-like jobs and as a replacement for the tentative getData proposal. See the reference documentation for how to migrate from getData to Datalink. * No longer using sesame's SOAP interface (that's been shut down by now) * gavo mkboost now takes the id of the directGrammar rather than a pair of rd id and and table id. It can now also parse from fits binary tables directly. * Datalink prototype changed during development: the parameter is now called ID in conformance with the WD; what was pubdid in the datalink functions is now called pubDID; several other changes that only concern you if you're running datalink (chances are you're not) * Any resource using \standardPubDID or similar needs to be re-imported. As that stuff hasn't been there in the last release, this probably doesn't concern you either. Version 0.8.9 (2013-07-13) * We now officially require postgres >9.0. If this hurts you, tell us, this is mainly a convenience matter. * DaCHS now supports the new BINARY2 VOTable encoding * Internal rework of web-facing parameter processing. If you use custom condDescs, test those services throughly (though we've tried to maintain backwards compatibility) * TAP examples are now DALI compliant * There's now a facility to add custom columns to the ObsCore table. * SIAP processing rework. If you have SIAP services, you should probably re-import them, since the old SIAP processing sometimes was a bit too lenient. * SIAP and SCS now count as "PQL" renderers, as SSAP before. This means custom parameters created via buildFrom will allow range and list syntaxes as for SSAP. * Many minor updates, fixes, and compatibility improvements. * New root-tree.html template that you can optionally use as the root page; it provides some ajaxy alternative to the old root page. Version 0.8.4 (2012-12-17) * New gavo upgrade command -- run this after every svn update or other upgrade (Debian packages to that automatically), and you should not have to worry about schema changes. * SIAP and SSAP dateObs columns are now MJD. Unless you have custom web interfaces or feed the tables by some other means than the setMeta proc applys, you should not notice (after a gavo upgrade). If your SIAP/SSAP tables are part of any views except ivoa.obscore, drop those views before running gavo upgrade, and check those services after upgrading, anyway. Version 0.8 (2012-08-15) * The schema for the DaCHS-internal metadata tables changed. To adapt your database, run gavo imp --system //dc_tables (to fill the tables again, say gavo pub -ma, but you'll do so in the next step anyway) * There's now a table tap_schema.groups. To create it, say: gavo imp //tap createSchema Do not give --system here unless you already have the tap_schema.groups table; if you do give --system, the existing tables will be re-created and you will have to say gavo pub -ma to re-fill them. * The schema of the service list has changed again, and the changes require a republication of all services. After the update, run gavo imp --system //services gavo pub -ma * Some ADQL user-defined functions are now implemented as Postgres stored procedures. Say:: gavo imp //adql define those. * The UWS code now writes to the UWS tables with less privileges. To let it do that, you must update the privileges on all queue tables; in standard DaCHS, that's only the TAP table, so simply say: gavo imp -m //tap createJobTable * The organization running the dc is now configurable. If you are running the registry interface, you must now give authority.managingOrg, organization.title, organization.description, and organization.referenceURL in defaultmeta.txt. authority.managingOrg usually is ivo:///org. * The former readRoles and allRoles attribute of table and resource elements are now called readProfiles and allProfiles and contain profile names rather than DB roles. In the unlikely case you're using this feature, you'd have to change the RDs. * You can no longer map db profile "roles" to actual profile names in the gavorc [profiles] section. If you actually need this, try symbolic links to the profile files. Version 0.7 (2012-01-12) * Automatic dependency management for published resources (register and adql). You must run gavo imp --system //services upgrade_0.6.3_0.7 after the update. * UWS job representation in the database changed. You must remove all UWS jobs before upgrading. After upgrading, run gavo imp //uws upgrade_0.6.3_0.7 gavo imp --system //tap createJobTable You can ignore error messages during the first of those commands.