Define measurements to be made and reported results from the Photometry Service

[talister]

From the Requirements Doc we have a basic definition of inputs and outputs for the Photometry Service. This issue serves to document these, along with any other additions that are needed, allowing determination of completion of the service.

Overview:

Abstraction of the service to perform low SNR source detection, source measurement and forced photometry at specified locations and return the results to the Controller and the User

Input Parameters:

• list of references to the images from Image Service to be measured (required) and either:
• (Source detection mode) :
• list (or columns) of the per-image uncertainty ellipse regions (required)
• SNR threshold (required)
• positional tolerance (required; some sensible default needed to override uncertainty for well determined objects)
• (Forced photometry mode):
• RA, Dec position pairs for each image (required)
• aperture radii to measure in (optional TBD if we let this be user configurable or fixed)

Output Parameters:

Either:

(Source detection mode)

• source catalog of detected objects and associated parameters that are within the uncertainty region and meet the positional and SNR thresholds
  or:
  (Forced photometry mode) table of photometry results containing at least:
• X/Y position (original detector coordinates and cutout coordinates)
• RA/Dec position
• aperture fluxes and errors in the specified apertures
• magnitude and error
• flags (for e.g. bad pixel proximity, other objects, incomplete apertures, saturation)
• centroiding uncertainty
• separation from known source (arcsec and sigma relative to covariance of ephemeris)

[hhsieh00]

@talister I'm not sure what centroiding uncertainty is for the output in forced photometry mode. Is this something like the distance between the peak pixel within a certain radius and the target coordinates (e.g., assuming there's a low S/N source close to the target coordinates)? If so, this would be different from my normal understanding of forced photometry, which is to simply measure the flux within a certain aperture at an exact position, in which case the "centroiding uncertainty" should always be zero? If the intention is for this code to also at least attempt source extraction in the vicinity of specified position, this should be clarified.

Also, "separation from known source" should probably be "separation from closest known source" or "separation from closest detected source", depending on what we want this to be, e.g., separation from the closest known source in the latest LSST static source catalog once one is available, or separation from the closest static source or DIA source found by the LSST pipeline for that specific image. In the latter case, this might depend on the use case, whether one is trying to determine whether a nearby detection might be the targeted object that is farther off its predicted position than expected for some reason (e.g., non-gravs or poor contributing astrometry to the current orbit solution), or if one is trying to determine the likelihood that the forced photometry measurement at a particular position could be contaminated by a nearby source, moving or otherwise. Just wanted to clarify your intention for this parameter...

[talister]

@talister I'm not sure what centroiding uncertainty is for the output in forced photometry mode. Is this something like the distance between the peak pixel within a certain radius and the target coordinates (e.g., assuming there's a low S/N source close to the target coordinates)? If so, this would be different from my normal understanding of forced photometry, which is to simply measure the flux within a certain aperture at an exact position, in which case the "centroiding uncertainty" should always be zero? If the intention is for this code to also at least attempt source extraction in the vicinity of specified position, this should be clarified.

Also, "separation from known source" should probably be "separation from closest known source" or "separation from closest detected source", depending on what we want this to be, e.g., separation from the closest known source in the latest LSST static source catalog once one is available, or separation from the closest static source or DIA source found by the LSST pipeline for that specific image. In the latter case, this might depend on the use case, whether one is trying to determine whether a nearby detection might be the targeted object that is farther off its predicted position than expected for some reason (e.g., non-gravs or poor contributing astrometry to the current orbit solution), or if one is trying to determine the likelihood that the forced photometry measurement at a particular position could be contaminated by a nearby source, moving or otherwise. Just wanted to clarify your intention for this parameter...

Hi @hhsieh00 , I think I was thinking of getting in a measure of how well the newly detected source was measured so something like Dave Tholen's semi-empirical HWHM/SNR formula for uncertainty on NEO positions. This feeds into @MarcBuie's comment/requirements that it would be good to know how sigma off the predicted position from the ephemeris the detected source was. It's all part of the post-query check on "is this a valid detection" or a coincidental chance alignment with an unrelated source.

[hhsieh00]

@talister, oh, okay, so the forced photometry is actually looking for a source. To me is more like low S/N source extraction, rather than forced photometry which I understand as "forcing" photometry at a specific spot, even if it's blank sky, but that could be fine...that's why I wanted clarification...but also just wanted to confirm that this is what we want.

For precovery, I think this makes sense, i.e., if there is a faint source in the uncertainty region that could be the object, that could give you the extra astrometry you want if it's actually the object. This would also probably be the desired functionality for phase function work (although the photometry will be highly uncertain if S/N is so low that it wasn't already picked up by the pipeline). In this case then, I think the output should include both the predicted x/y and measured x/y of the detected source.

That said, for outburst detection (e.g., looking for objects that were detected in recent data that should be visible based on simple phase function assumptions at previous pointings where they were not detected, possibly because they were faint at the earlier time, but underwent recent outburst events that pushed them into the detectable range in more recent data), people would probably want to know what the flux is at specific predicted positions (depending on how large the uncertainty region is), regardless of whether there is a source there is not (i.e., to either get photometry of a faint source, if present, or the detection limit if no source is visible at all). Would it be possible to do both?

[MarcBuie]

I added to two output quantities to better articulate what's desired. This is consistent with discussion on the call. Regarding (x,y) location, is having just the original detector coordinates enough? For the intended use, I would think you also need to know the detector ID. If you don't (or can't) know the detector, I don't think the absolute coordinates tell you anything unless it's in some sort of absolute focal plane coordinate (which wouldn't be in pixels).

For the "distance" from the ephemeris output, both quantities I put down are useful, but the sigma measure is much, much more useful.

For input quantities, there is an SNR threshold that attempts to pre-filter the output. That's fine. But if this is useful, wouldn't it also be useful to have an optional threshold on sigma "distance" from the ephemeris? As the ellipses flatten out with increasing quality of the fitted orbit, sources in the minor-axis direction will get filtered out much more easily. This approach isn't a rigorously applied in the community as it should be, but maybe this will help.

[szilac]

Current state of the project

(image_photometry branch)

1. Image Service Functionality
The image service now performs searches in the ObsCore database based on pre-downloaded ephemeris data. Initially, the search filters the data using the start and end times of the entire ephemeris dataset. Subsequently, additional filters are applied.
The query refines the search using time intervals and coordinates derived from the boundaries of individual ephemeris datasets.

2. ImageMetadata dataclass
The image service generates an ImageMetadata dataclass for each search result. This dataclass contains:
- visit_id, detector_id, ccdvisit, and band, coordinates_central, t_min and t_max
- ephemeris_data (a list of the relevant ephemeris entries)
- Exact_ephemeris (interpolated coordinates derived from the ephemeris dataset)
- The ImageMetadata dataclass serves as the primary input for the photometry service.

3. Photometry Service Workflow
The photometry service operates in the following steps:
- Downloads the relevant image and performs a cutout process.
- Conducts source detection within the cutout image.
- Checks if any sources are located within the defined error ellipse.
- Performs forced photometry at the target coordinates regardless of the presence of detected sources.

4. EndResult dataclass
Following the photometry process, an EndResults dataclass is created. It includes:
- Metadata about the target and the corresponding image
- Results of the forced photometry
- Photometric data of all detected sources within the error ellipse, organized as a list

5. Final output
- The EndResults dataclass is saved as a JSON file, which serves as the final output of the process.

Challenges to Address

1. Scalability Issues with Image Service
The image service encounters failures when processing ephemeris datasets with more than 500 rows. This issue arises because the generated query includes conditions for every ephemeris row, such as:
- t-1 < t_obs < t+1
- CONTAINS(POINT('ICRS', {ra}, {dec}), s_region) = 1
- For large datasets, this results in an excessively long query, which overwhelms the SQL server.
2. Current DP0.2 Limitations
The current DP0.2 database does not support uploading tables or data files to the SQL server for performing searches using JOIN operations. This would be the ideal solution to manage large ephemeris datasets efficiently.
However, DP0.3 is already include this feature but I don't know when will be available for a relevant database.

Potential solution

1. Performing Local Searches
The ObsCore catalog contains approximately 2.8 million rows (~500MB), making it challenging to handle locally. Tests are ongoing to evaluate feasible methods for managing and querying the catalog efficiently on local systems.

2. Awaiting SQL Server Enhancements
Delaying this feature until the next data release as its expanded SQL server functionality could resolve the issue without significant changes to the existing workflow.

3. Dividing the Search into Smaller Time Segments
Splitting the search into smaller time intervals could mitigate the query length issue, allowing the current system to handle the searches but the runtime will be significant.

I am open to and would greatly appreciate any ideas or methods that could help resolve this issue.

[szilac]

I tried to follow up on the solutions that came up in the last meeting.

Currently you can search for a point within a polygon, but finding the intersection of 2 polygons is not implemented yet on the server.
Because of this, there will be no elegant solution to the problem at this time. The best I can do is to split the input dataset into manageable sized chunks and search that way.

Currently in the process of being realized.