-
Notifications
You must be signed in to change notification settings - Fork 9
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #105 from pdowler/dap-proto
DAP prototype
- Loading branch information
Showing
69 changed files
with
4,749 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
FROM images.opencadc.org/library/cadc-tomcat:1 | ||
|
||
COPY build/libs/dap.war /usr/share/tomcat/webapps |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,87 @@ | ||
# dap | ||
|
||
`dap` is a prototype [Data Access Protocol](https://github.com/ivoa-std/DAP) service | ||
that should work with any TAP service that provides an ivoa.ObsCore table (or view). It | ||
also supports a mode that makes it operate as a compliant | ||
[Simple Image Access 2.0](https://www.ivoa.net/documents/SIA/) service. | ||
|
||
## deployment | ||
The `dap` war file can be renamed at deployment time in order to support an | ||
alternate service name, including introducing additional path elements using the | ||
[war-rename.conf](https://github.com/opencadc/docker-base/tree/master/cadc-tomcat) | ||
feature. | ||
|
||
This service instance is expected to have a PostgreSQL database backend to store UWS | ||
job information. This requirement could be removed in future to support a more lightweight | ||
deployment of what is essentially a facade on a TAP service. | ||
|
||
## configuration | ||
The following configuration files must be available in the `/config` directory. | ||
|
||
### catalina.properties | ||
This file contains java system properties to configure the tomcat server and some of the java | ||
libraries used in the service. | ||
|
||
See <a href="https://github.com/opencadc/docker-base/tree/master/cadc-tomcat">cadc-tomcat</a> | ||
for system properties related to the deployment environment. | ||
|
||
See <a href="https://github.com/opencadc/core/tree/master/cadc-util">cadc-util</a> | ||
for common system properties. | ||
|
||
`dap` includes multiple IdentityManager implementations to support authenticated access: | ||
- See <a href="https://github.com/opencadc/ac/tree/master/cadc-access-control-identity">cadc-access-control-identity</a> for CADC access-control system support. | ||
- See <a href="https://github.com/opencadc/ac/tree/master/cadc-gms">cadc-gms</a> for OIDC token support. | ||
|
||
`dap` requires one connection pool to store jobs: | ||
``` | ||
# database connection pools | ||
org.opencadc.dap.uws.maxActive={max connections for jobs pool} | ||
org.opencadc.dap.uws.username={database username for jobs pool} | ||
org.opencadc.dap.uws.password={database password for jobs pool} | ||
org.opencadc.dap.uws.url=jdbc:postgresql://{server}/{database} | ||
``` | ||
The _uws_ pool manages (create, alter, drop) uws tables and manages the uws content | ||
(creates and modifies jobs in the uws schema when jobs are created and executed by users. | ||
|
||
### cadc-registry.properties | ||
See <a href="https://github.com/opencadc/reg/tree/master/cadc-registry">cadc-registry</a>. | ||
|
||
### dap.properties | ||
`dap` must be configured to use a single TAP service to execute queries. | ||
``` | ||
# TAP service | ||
org.opencadc.dap.queryService = {resourceID or TAP base URL} | ||
# run in backwards compatible SIAv2 mode (optional) | ||
org.opencadc.dap.sia2mode = true | false | ||
# use a specific ObsCore table in the query service (optional, default: ivoa.ObsCore) | ||
org.opencadc.dap.table = {table name} | ||
``` | ||
The _queryService_ is resolved by a registry lookup and that service is used to query | ||
for CAOM content. It is assumed that this service is deployed "locally" since there can | ||
be many calls and low latency is very desireable. | ||
|
||
The _sia2mode_ can be set to make the service behave as an SIA-2.0 service: this causes | ||
the generated query to restrict the ObsCore.dataproduct_type values to `cube` and `image`. | ||
|
||
The _table_ to query can be set to something other than the default (`ivoa.ObsCore`). | ||
|
||
`dap` will attempt to use the caller's identity to query, but the details of this depend | ||
on the configured IdentityManager and local A&A service configuration. | ||
|
||
## building it | ||
``` | ||
gradle clean build | ||
docker build -t dap -f Dockerfile . | ||
``` | ||
|
||
## checking it | ||
``` | ||
docker run --rm -it dap:latest /bin/bash | ||
``` | ||
|
||
## running it | ||
``` | ||
docker run --rm --user tomcat:tomcat --volume=/path/to/external/config:/config:ro --name dap dap:latest | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
## deployable containers have a semantic and build tag | ||
# semantic version tag: major.minor | ||
# build version tag: timestamp | ||
VER=0.1.0 | ||
TAGS="${VER} ${VER}-$(date -u +"%Y%m%dT%H%M%S")" | ||
unset VER |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,54 @@ | ||
plugins { | ||
id 'maven' | ||
id 'war' | ||
id 'checkstyle' | ||
} | ||
|
||
repositories { | ||
mavenCentral() | ||
mavenLocal() | ||
} | ||
|
||
apply from: '../opencadc.gradle' | ||
|
||
sourceCompatibility = 11 | ||
|
||
group = 'ca.nrc.cadc' | ||
|
||
war { | ||
// Include the swagger-ui so that /sia provides the Sia API documentation | ||
from(System.getenv('RPS') + '/resources/') { | ||
include 'swagger-ui/' | ||
} | ||
from('.') { | ||
include 'VERSION' | ||
} | ||
} | ||
|
||
dependencies { | ||
providedCompile 'javax.servlet:javax.servlet-api:[3.1.0,)' | ||
|
||
compile 'org.opencadc:cadc-util:[1.6.1,)' | ||
compile 'org.opencadc:cadc-cdp:[1.2.3,)' | ||
compile 'org.opencadc:cadc-uws-server:[1.2.4,)' | ||
compile 'org.opencadc:cadc-tap:[1.0,2.0)' | ||
compile 'org.opencadc:cadc-vosi:[1.4.3,2.0)' | ||
|
||
runtime 'org.opencadc:cadc-registry:[1.7.7,)' | ||
runtime 'org.opencadc:cadc-log:[1.0,)' | ||
runtime 'org.opencadc:cadc-gms:[1.0.7,2.0)' | ||
runtime 'org.opencadc:cadc-access-control-identity:[1.1.0,)' | ||
|
||
testCompile 'junit:junit:[4.0,)' | ||
|
||
intTestCompile 'org.opencadc:cadc-test-vosi:[1.0.11,)' | ||
intTestCompile 'org.opencadc:cadc-test-uws:[1.1,)' | ||
} | ||
|
||
configurations { | ||
runtime.exclude group: 'javax.servlet' | ||
runtime.exclude group: 'net.sourceforge.jtds' | ||
runtime.exclude group: 'org.postgresql' | ||
runtime.exclude group: 'org.restlet.jee' | ||
} | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,85 @@ | ||
/* | ||
************************************************************************ | ||
******************* CANADIAN ASTRONOMY DATA CENTRE ******************* | ||
************** CENTRE CANADIEN DE DONNÉES ASTRONOMIQUES ************** | ||
* | ||
* (c) 2024. (c) 2024. | ||
* Government of Canada Gouvernement du Canada | ||
* National Research Council Conseil national de recherches | ||
* Ottawa, Canada, K1A 0R6 Ottawa, Canada, K1A 0R6 | ||
* All rights reserved Tous droits réservés | ||
* | ||
* NRC disclaims any warranties, Le CNRC dénie toute garantie | ||
* expressed, implied, or énoncée, implicite ou légale, | ||
* statutory, of any kind with de quelque nature que ce | ||
* respect to the software, soit, concernant le logiciel, | ||
* including without limitation y compris sans restriction | ||
* any warranty of merchantability toute garantie de valeur | ||
* or fitness for a particular marchande ou de pertinence | ||
* purpose. NRC shall not be pour un usage particulier. | ||
* liable in any event for any Le CNRC ne pourra en aucun cas | ||
* damages, whether direct or être tenu responsable de tout | ||
* indirect, special or general, dommage, direct ou indirect, | ||
* consequential or incidental, particulier ou général, | ||
* arising from the use of the accessoire ou fortuit, résultant | ||
* software. Neither the name de l'utilisation du logiciel. Ni | ||
* of the National Research le nom du Conseil National de | ||
* Council of Canada nor the Recherches du Canada ni les noms | ||
* names of its contributors may de ses participants ne peuvent | ||
* be used to endorse or promote être utilisés pour approuver ou | ||
* products derived from this promouvoir les produits dérivés | ||
* software without specific prior de ce logiciel sans autorisation | ||
* written permission. préalable et particulière | ||
* par écrit. | ||
* | ||
* This file is part of the Ce fichier fait partie du projet | ||
* OpenCADC project. OpenCADC. | ||
* | ||
* OpenCADC is free software: OpenCADC est un logiciel libre ; | ||
* you can redistribute it and/or vous pouvez le redistribuer ou le | ||
* modify it under the terms of modifier suivant les termes de | ||
* the GNU Affero General Public la “GNU Affero General Public | ||
* License as published by the License” telle que publiée | ||
* Free Software Foundation, par la Free Software Foundation | ||
* either version 3 of the : soit la version 3 de cette | ||
* License, or (at your option) licence, soit (à votre gré) | ||
* any later version. toute version ultérieure. | ||
* | ||
* OpenCADC is distributed in the OpenCADC est distribué | ||
* hope that it will be useful, dans l’espoir qu’il vous | ||
* but WITHOUT ANY WARRANTY; sera utile, mais SANS AUCUNE | ||
* without even the implied GARANTIE : sans même la garantie | ||
* warranty of MERCHANTABILITY implicite de COMMERCIALISABILITÉ | ||
* or FITNESS FOR A PARTICULAR ni d’ADÉQUATION À UN OBJECTIF | ||
* PURPOSE. See the GNU Affero PARTICULIER. Consultez la Licence | ||
* General Public License for Générale Publique GNU Affero | ||
* more details. pour plus de détails. | ||
* | ||
* You should have received Vous devriez avoir reçu une | ||
* a copy of the GNU Affero copie de la Licence Générale | ||
* General Public License along Publique GNU Affero avec | ||
* with OpenCADC. If not, see OpenCADC ; si ce n’est | ||
* <http://www.gnu.org/licenses/>. pas le cas, consultez : | ||
* <http://www.gnu.org/licenses/>. | ||
* | ||
************************************************************************ | ||
*/ | ||
|
||
package org.opencadc.dap; | ||
|
||
import java.net.URI; | ||
import org.apache.log4j.Logger; | ||
|
||
/** | ||
* | ||
* @author pdowler | ||
*/ | ||
public class Constants { | ||
private static final Logger log = Logger.getLogger(Constants.class); | ||
|
||
public static URI DAP_RESOURCE_ID = URI.create("ivo://opencadc.org/dap"); | ||
public static URI SIA_RESOURCE_ID = URI.create("ivo://opencadc.org/sia"); | ||
|
||
private Constants() { | ||
} | ||
} |
127 changes: 127 additions & 0 deletions
127
dap/src/intTest/java/org/opencadc/dap/DapQueryErrorTest.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,127 @@ | ||
/* | ||
************************************************************************ | ||
******************* CANADIAN ASTRONOMY DATA CENTRE ******************* | ||
************** CENTRE CANADIEN DE DONNÉES ASTRONOMIQUES ************** | ||
* | ||
* (c) 2023. (c) 2023. | ||
* Government of Canada Gouvernement du Canada | ||
* National Research Council Conseil national de recherches | ||
* Ottawa, Canada, K1A 0R6 Ottawa, Canada, K1A 0R6 | ||
* All rights reserved Tous droits réservés | ||
* | ||
* NRC disclaims any warranties, Le CNRC dénie toute garantie | ||
* expressed, implied, or énoncée, implicite ou légale, | ||
* statutory, of any kind with de quelque nature que ce | ||
* respect to the software, soit, concernant le logiciel, | ||
* including without limitation y compris sans restriction | ||
* any warranty of merchantability toute garantie de valeur | ||
* or fitness for a particular marchande ou de pertinence | ||
* purpose. NRC shall not be pour un usage particulier. | ||
* liable in any event for any Le CNRC ne pourra en aucun cas | ||
* damages, whether direct or être tenu responsable de tout | ||
* indirect, special or general, dommage, direct ou indirect, | ||
* consequential or incidental, particulier ou général, | ||
* arising from the use of the accessoire ou fortuit, résultant | ||
* software. Neither the name de l'utilisation du logiciel. Ni | ||
* of the National Research le nom du Conseil National de | ||
* Council of Canada nor the Recherches du Canada ni les noms | ||
* names of its contributors may de ses participants ne peuvent | ||
* be used to endorse or promote être utilisés pour approuver ou | ||
* products derived from this promouvoir les produits dérivés | ||
* software without specific prior de ce logiciel sans autorisation | ||
* written permission. préalable et particulière | ||
* par écrit. | ||
* | ||
* This file is part of the Ce fichier fait partie du projet | ||
* OpenCADC project. OpenCADC. | ||
* | ||
* OpenCADC is free software: OpenCADC est un logiciel libre ; | ||
* you can redistribute it and/or vous pouvez le redistribuer ou le | ||
* modify it under the terms of modifier suivant les termes de | ||
* the GNU Affero General Public la “GNU Affero General Public | ||
* License as published by the License” telle que publiée | ||
* Free Software Foundation, par la Free Software Foundation | ||
* either version 3 of the : soit la version 3 de cette | ||
* License, or (at your option) licence, soit (à votre gré) | ||
* any later version. toute version ultérieure. | ||
* | ||
* OpenCADC is distributed in the OpenCADC est distribué | ||
* hope that it will be useful, dans l’espoir qu’il vous | ||
* but WITHOUT ANY WARRANTY; sera utile, mais SANS AUCUNE | ||
* without even the implied GARANTIE : sans même la garantie | ||
* warranty of MERCHANTABILITY implicite de COMMERCIALISABILITÉ | ||
* or FITNESS FOR A PARTICULAR ni d’ADÉQUATION À UN OBJECTIF | ||
* PURPOSE. See the GNU Affero PARTICULIER. Consultez la Licence | ||
* General Public License for Générale Publique GNU Affero | ||
* more details. pour plus de détails. | ||
* | ||
* You should have received Vous devriez avoir reçu une | ||
* a copy of the GNU Affero copie de la Licence Générale | ||
* General Public License along Publique GNU Affero avec | ||
* with OpenCADC. If not, see OpenCADC ; si ce n’est | ||
* <http://www.gnu.org/licenses/>. pas le cas, consultez : | ||
* <http://www.gnu.org/licenses/>. | ||
* | ||
* $Revision: 5 $ | ||
* | ||
************************************************************************ | ||
*/ | ||
|
||
package org.opencadc.dap; | ||
|
||
import ca.nrc.cadc.conformance.uws2.JobResultWrapper; | ||
import ca.nrc.cadc.conformance.uws2.SyncUWSTest; | ||
import ca.nrc.cadc.dali.tables.votable.VOTableDocument; | ||
import ca.nrc.cadc.reg.Standards; | ||
import ca.nrc.cadc.util.FileUtil; | ||
import ca.nrc.cadc.util.Log4jInit; | ||
import java.io.File; | ||
import java.net.URI; | ||
import org.apache.log4j.Level; | ||
import org.apache.log4j.Logger; | ||
import org.junit.Assert; | ||
|
||
/** | ||
* | ||
* @author pdowler | ||
*/ | ||
public class DapQueryErrorTest extends SyncUWSTest { | ||
|
||
private static final Logger log = Logger.getLogger(DapQueryErrorTest.class); | ||
|
||
static { | ||
Log4jInit.setLevel("org.opencadc.dap", Level.INFO); | ||
Log4jInit.setLevel("ca.nrc.cadc.conformance.uws2", Level.INFO); | ||
} | ||
|
||
public DapQueryErrorTest() { | ||
super(Constants.DAP_RESOURCE_ID, Standards.DAP_QUERY_21); | ||
|
||
File testFile = FileUtil.getFileFromResource("SyncTest-ERROR-BAND.properties", DapQueryErrorTest.class); | ||
if (testFile.exists()) { | ||
File testDir = testFile.getParentFile(); | ||
super.setPropertiesDir(testDir, "SyncTest-ERROR"); | ||
} | ||
} | ||
|
||
@Override | ||
protected void validateResponse(JobResultWrapper result) { | ||
Assert.assertEquals(400, result.responseCode); | ||
Assert.assertEquals("application/x-votable+xml", result.contentType); | ||
|
||
try { | ||
Assert.assertNotNull(result.syncOutput); | ||
//VOTableDocument vot = VOTableHandler.getVOTable(result.syncOutput); | ||
// because cadc-util HttpTransfer reads the error body and stores it in the Exception | ||
VOTableDocument vot = VOTableHandler.getVOTable(result.throwable); | ||
log.info(result.name + ": found valid VOTable"); | ||
|
||
String queryStatus = VOTableHandler.getQueryStatus(vot); | ||
Assert.assertNotNull("QUERY_STATUS", queryStatus); | ||
Assert.assertEquals("ERROR", queryStatus); | ||
} catch (Exception ex) { | ||
log.error("unexpected exception", ex); | ||
Assert.fail("unexpected exception: " + ex); | ||
} | ||
} | ||
} |
Oops, something went wrong.