Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue 693 - Rating curve retrieval #909

Open
wants to merge 5 commits into
base: develop
Choose a base branch
from
Open

Issue 693 - Rating curve retrieval #909

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
142fc4f
Issue 693 - Update to allow finding closest effective date in progress
zack-rma Sep 30, 2024
1f3a956
Issue 693 - Retrieve most recent rating
zack-rma Oct 10, 2024
6d92679
Issue 693 - Updated recent rating retrieval, improved test cases
zack-rma Oct 11, 2024
e300f64
Issue 693 - Updated effectiveDate variable naming
zack-rma Oct 11, 2024
412716e
Rewrote closest date finder query, removed logic from CDA
zack-rma Oct 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 25 additions & 16 deletions cwms-data-api/src/main/java/cwms/cda/api/RatingController.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
import static cwms.cda.api.Controllers.DATE_FORMAT;
import static cwms.cda.api.Controllers.DATUM;
import static cwms.cda.api.Controllers.DELETE;
import static cwms.cda.api.Controllers.EFFECTIVE_DATE;
import static cwms.cda.api.Controllers.END;
import static cwms.cda.api.Controllers.EXAMPLE_DATE;
import static cwms.cda.api.Controllers.FORMAT;
Expand Down Expand Up @@ -296,19 +297,16 @@ public void getAll(@NotNull Context ctx) {

ContentType contentType = Formats.parseHeaderAndQueryParm(header, format, RatingAliasMarker.class);

if (format.isEmpty())
{
if (format.isEmpty()) {
//Use the full content type here (i.e. application/json;version=2)
ctx.contentType(contentType.toString());
}
else
{
} else {
//Legacy content type only includes the basic type (i.e. application/json)
ctx.contentType(contentType.getType());
}

//At the moment, we still use the legacy formatting here, since we don't have a newer API for serializing/deserializing
//a collection of rating sets - unlike getOne.
//At the moment, we still use the legacy formatting here, since we don't have a newer API for
// serializing/deserializing a collection of rating sets - unlike getOne.
String legacyFormat = Formats.getLegacyTypeFromContentType(contentType);
String results = ratingDao.retrieveRatings(legacyFormat, names, unit, datum, office, start,
end, timezone);
Expand All @@ -322,7 +320,8 @@ public void getAll(@NotNull Context ctx) {

@OpenApi(
pathParams = {
@OpenApiParam(name = RATING_ID, required = true, description = "The rating-id of the effective dates to be retrieve. "),
@OpenApiParam(name = RATING_ID, required = true, description = "The rating-id of the effective "
+ "dates to be retrieve. "),
},
queryParams = {
@OpenApiParam(name = OFFICE, required = true, description =
Expand All @@ -339,6 +338,10 @@ public void getAll(@NotNull Context ctx) {
+ "otherwise specified), as well as the time zone of any times in the"
+ " response. If this field is not specified, the default time zone "
+ "of UTC shall be used."),
@OpenApiParam(name = EFFECTIVE_DATE, description = "Specifies the "
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a very confusing parameter.

  • three optional date parameters are confusing
  • what happens when all three are input?
  • the ask was for the current effective date, that means the client needs to always set this parameter to now. Why not just have another endpoint that does it automatically? /{rating-id}/latest-effective
  • the "closest" effective date shouldn't return a newer effective date than what is supplied as the newer effective date is not yet in effect, though this is made more complicated by temporal interpolations and transition dates.
  • how do active ratings play an effect? I presume they would be skipped, but that's missing from the docs (even from the start/end parameters)

Additionally, you are passing this confusion into the DAO by expanding the existing DAO method instead of creating a new one, so now you have three dates in the DAO function. This makes test cases far more complicated.

+ "date to find the closest match to for retrieving a specific rating curve. This date is used "
+ "instead of the time window specified by start and end. "
+ "The format for this field is ISO 8601 extended."),
@OpenApiParam(name = METHOD, description = "Specifies "
+ "the retrieval method used. If no method is provided EAGER will be used.",
type = RatingSet.DatabaseLoadMethod.class),
Expand All @@ -354,8 +357,13 @@ public void getAll(@NotNull Context ctx) {
public void getOne(@NotNull Context ctx, @NotNull String rating) {

try (final Timer.Context ignored = markAndTime(GET_ONE)) {
String officeId = ctx.queryParam(OFFICE);

String timezone = ctx.queryParamAsClass(TIMEZONE, String.class).getOrDefault("UTC");
Instant effectiveDate = null;
String effectiveDateParam = ctx.queryParam(EFFECTIVE_DATE);
if (effectiveDateParam != null) {
effectiveDate = DateUtils.parseUserDate(effectiveDateParam, timezone).toInstant();
}

Instant beginInstant = null;
String begin = ctx.queryParam(BEGIN);
Expand All @@ -369,11 +377,13 @@ public void getOne(@NotNull Context ctx, @NotNull String rating) {
endInstant = DateUtils.parseUserDate(end, timezone).toInstant();
}

String officeId = ctx.queryParam(OFFICE);

RatingSet.DatabaseLoadMethod method = ctx.queryParamAsClass(METHOD,
RatingSet.DatabaseLoadMethod.class)
.getOrDefault(RatingSet.DatabaseLoadMethod.EAGER);

String body = getRatingSetString(ctx, method, officeId, rating, beginInstant, endInstant);
String body = getRatingSetString(ctx, method, officeId, rating, beginInstant, endInstant, effectiveDate);
if (body != null) {
ctx.result(body);
ctx.status(HttpCode.OK);
Expand All @@ -385,7 +395,7 @@ public void getOne(@NotNull Context ctx, @NotNull String rating) {
@Nullable
private String getRatingSetString(Context ctx, RatingSet.DatabaseLoadMethod method,
String officeId, String rating, Instant begin,
Instant end) {
Instant end, Instant effectiveDate) {
String retval = null;

try (final Timer.Context ignored = markAndTime("getRatingSetString")) {
Expand All @@ -398,7 +408,7 @@ private String getRatingSetString(Context ctx, RatingSet.DatabaseLoadMethod meth
if (isJson || isXml) {
ctx.contentType(contentType.toString());
try {
RatingSet ratingSet = getRatingSet(ctx, method, officeId, rating, begin, end);
RatingSet ratingSet = getRatingSet(ctx, method, officeId, rating, begin, end, effectiveDate);
if (ratingSet != null) {
if (isJson) {
retval = JsonRatingUtils.toJson(ratingSet);
Expand All @@ -424,8 +434,7 @@ private String getRatingSetString(Context ctx, RatingSet.DatabaseLoadMethod meth
} else {
CdaError re = new CdaError("Currently supporting only: " + Formats.JSONV2
+ " and " + Formats.XMLV2);
logger.log(Level.WARNING, "Provided accept header not recognized:"
+ acceptHeader, re);
logger.log(Level.WARNING, String.format("Provided accept header not recognized: %s", acceptHeader), re);
ctx.status(HttpServletResponse.SC_NOT_IMPLEMENTED);
ctx.json(CdaError.notImplemented());
}
Expand All @@ -437,13 +446,13 @@ private String getRatingSetString(Context ctx, RatingSet.DatabaseLoadMethod meth

private RatingSet getRatingSet(Context ctx, RatingSet.DatabaseLoadMethod method,
String officeId, String rating, Instant begin,
Instant end) throws IOException, RatingException {
Instant end, Instant effectiveDate) throws IOException, RatingException {
RatingSet ratingSet;
try (final Timer.Context ignored = markAndTime("getRatingSet")) {
DSLContext dsl = getDslContext(ctx);

RatingDao ratingDao = getRatingDao(dsl);
ratingSet = ratingDao.retrieve(method, officeId, rating, begin, end);
ratingSet = ratingDao.retrieve(method, officeId, rating, begin, end, effectiveDate);
}

return ratingSet;
Expand Down
4 changes: 2 additions & 2 deletions cwms-data-api/src/main/java/cwms/cda/data/dao/RatingDao.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,12 @@

public interface RatingDao {

static final Pattern officeMatcher = Pattern.compile(".*office-id=\"(.*?)\"");
Pattern officeMatcher = Pattern.compile(".*office-id=\"(.*?)\"");

void create(String ratingSet, boolean storeTemplate) throws IOException, RatingException;

RatingSet retrieve(RatingSet.DatabaseLoadMethod method, String officeId, String specificationId,
Instant start, Instant end) throws IOException, RatingException;
Instant start, Instant end, Instant effectiveDate) throws IOException, RatingException;

String retrieveRatings(String format, String names, String unit, String datum, String office,
String start, String end, String timezone);
Expand Down
109 changes: 93 additions & 16 deletions cwms-data-api/src/main/java/cwms/cda/data/dao/RatingSetDao.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,22 +24,42 @@

package cwms.cda.data.dao;

import static org.jooq.impl.DSL.abs;
import static org.jooq.impl.DSL.asterisk;
import static org.jooq.impl.DSL.dateDiff;
import static org.jooq.impl.DSL.field;
import static org.jooq.impl.DSL.inline;
import static org.jooq.impl.DSL.name;
import static org.jooq.impl.DSL.partitionBy;
import static org.jooq.impl.DSL.rank;
import static org.jooq.impl.DSL.select;
import static org.jooq.impl.DSL.table;
import static org.jooq.impl.DSL.toDate;
import static org.jooq.impl.DSL.unquotedName;

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.dataformat.xml.XmlMapper;
import hec.data.RatingException;
import hec.data.cwmsRating.RatingSet;
import java.io.IOException;
import java.sql.Connection;
import java.sql.Date;
import java.sql.Timestamp;
import java.time.Instant;
import java.time.ZoneId;
import java.time.format.DateTimeFormatter;
import java.util.List;
import mil.army.usace.hec.cwms.rating.io.jdbc.ConnectionProvider;
import mil.army.usace.hec.cwms.rating.io.jdbc.RatingJdbcFactory;
import org.jooq.DSLContext;
import org.jooq.DatePart;
import org.jooq.Name;
import org.jooq.Record;
import org.jooq.exception.DataAccessException;
import usace.cwms.db.jooq.codegen.packages.CWMS_RATING_PACKAGE;
import usace.cwms.db.jooq.codegen.tables.AV_RATING_LOCAL;

import java.io.IOException;
import java.sql.Connection;
import java.sql.Timestamp;
import java.time.Instant;
import java.util.List;

public class RatingSetDao extends JooqDao<RatingSet> implements RatingDao {

Expand All @@ -57,8 +77,7 @@ public void create(String ratingSetXml, boolean storeTemplate) throws IOExceptio
DSLContext context = getDslContext(c, office);
String errs = CWMS_RATING_PACKAGE.call_STORE_RATINGS_XML__5(context.configuration(),
ratingSetXml, "T", storeTemplate ? "T" : "F");
if (errs != null && !errs.isEmpty())
{
if (errs != null && !errs.isEmpty()) {
throw new DataAccessException(errs);
}
});
Expand All @@ -85,8 +104,9 @@ private static String extractOfficeId(String ratingSet) throws JsonProcessingExc

@Override
public RatingSet retrieve(RatingSet.DatabaseLoadMethod method, String officeId,
String specificationId, Instant startZdt, Instant endZdt
String specificationId, Instant startZdt, Instant endZdt, Instant effectiveDateParam
) throws IOException, RatingException {
AV_RATING_LOCAL view = AV_RATING_LOCAL.AV_RATING_LOCAL;

final RatingSet[] retval = new RatingSet[1];
try {
Expand All @@ -110,9 +130,66 @@ public RatingSet retrieve(RatingSet.DatabaseLoadMethod method, String officeId,

RatingSet.DatabaseLoadMethod finalMethod = method;

connection(dsl, c -> retval[0] =
RatingJdbcFactory.ratingSet(finalMethod, new RatingConnectionProvider(c), officeId,
specificationId, start, end, false));
if (effectiveDateParam != null) {
ZoneId utcZone = ZoneId.of("UTC");
Name rank = unquotedName("rank");
Name alias1 = unquotedName("a");
Name alias2 = unquotedName("b");
Name alias3 = unquotedName("c");
Name diff = unquotedName("difference");
Name effectiveDateName = unquotedName("effective_date");
Name ratingIdName = unquotedName("rating_id");
String dateFormat = "DD-MMM-YYYY H:mm:ss";
String oracleDateFormat = "DD-MON-YYYY HH24:MI:SS";
DateTimeFormatter formatter = DateTimeFormatter.ofPattern(dateFormat)
.withZone(utcZone);
String formattedDate = formatter.format(effectiveDateParam);

String ratingId = null;
Instant startInstant = null;
Instant endInstant = null;
Record result = connectionResult(dsl, c ->
dsl.select(asterisk())
.from(
select(
table(alias1).asterisk(),
rank().over(partitionBy(field(alias3))
.orderBy(field(diff).asc()))
.as(rank)
)
.from(
select(
table(alias2).asterisk(),
field(name(alias2, ratingIdName)).as(alias3),
abs(dateDiff(DatePart.DAY, toDate(formattedDate, inline(oracleDateFormat)),
field(name(alias2, effectiveDateName))
.cast(Date.class)))
.as(diff))
.from(view.as(alias2))
.asTable(alias1)))
.where(field(rank).eq(inline(1)))
.fetchOne()
);
Timestamp effectiveDate = result.getValue(view.EFFECTIVE_DATE, Timestamp.class);
ratingId = result.getValue(view.RATING_ID);
startInstant = effectiveDate.toInstant().atZone(utcZone).toInstant();
endInstant = effectiveDate.toInstant().atZone(utcZone).toInstant();

if (ratingId == null) {
return null;
}
final Instant finalStartInstant = startInstant;
final String ratingIdFinal = ratingId;
final Instant finalEndInstant = endInstant;
connection(dsl, c -> retval[0] =
RatingJdbcFactory.ratingSet(finalMethod, new RatingConnectionProvider(c), officeId,
ratingIdFinal, finalStartInstant.toEpochMilli(),
finalEndInstant.toEpochMilli(), false));
} else {
connection(dsl, c -> retval[0] =
RatingJdbcFactory.ratingSet(finalMethod, new RatingConnectionProvider(c), officeId,
specificationId, start, end, false));
}

} catch (DataAccessException ex) {
Throwable cause = ex.getCause();
Expand Down Expand Up @@ -151,7 +228,7 @@ public void store(String ratingSetXml, boolean includeTemplate) throws IOExcepti
public void delete(String officeId, String specificationId, Instant start, Instant end) {
Timestamp startDate = new Timestamp(start.toEpochMilli());
Timestamp endDate = new Timestamp(end.toEpochMilli());
dsl.connection(c->
dsl.connection(c ->
CWMS_RATING_PACKAGE.call_DELETE_RATINGS(
getDslContext(c,officeId).configuration(), specificationId, startDate,
endDate, "UTC", officeId
Expand All @@ -170,15 +247,15 @@ public String retrieveRatings(String format, String names, String unit, String d
}

private static final class RatingConnectionProvider implements ConnectionProvider {
private final Connection c;
private final Connection conn;

private RatingConnectionProvider(Connection c) {
this.c = c;
private RatingConnectionProvider(Connection conn) {
this.conn = conn;
}

@Override
public Connection getConnection() {
return c;
return conn;
}

@Override
Expand Down
Loading
Loading