Starcamera - first PR request for testing starcamera at the LAT #335

karenperezsarmiento · 2022-08-01T21:08:39Z

Description

The latest version of the starcamera agent is able to connect to the device, receive telemetry/astrometry data and save it to the correct registries.

Motivation and Context

Agent is capable to connecting to device and receiving telemetry data.

How Has This Been Tested?

The agent has been tested with the starcamera system connected but dark (test run not done on real sky).

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
Unless I am preparing a release, I have opened this PR onto the develop branch.

sanahabhimani

Thanks for the PR, Karen! Sorry it took us a while to get back to you; since your initial request, we've restructured socs so PR's are now made to merge into main instead of develop.

I changed a lot of the key structure of your star camera agent branch to make it compatible with the updated socs. This way, the comments are more related to the agent itself. As a result, when you go to make updates to this agent, be sure to pull from the starcamera branch first to make sure you have the up-to-date version of this branch.

A couple notes were made on the extra files pushed to the branch.

One major note that isn't present in the comments below: you'll also need to write documentation for your agent so that it's available to see on the socs readthedocs page.

sanahabhimani · 2023-05-25T17:21:56Z

socs/agents/starcam/agent.py

+from ocs.ocs_twisted import TimeoutLock
+
+
+class starcam_Helper:


class name should follow pep8 rules; i.e., StarcamHelper. Same goes for the agent's class name. Once those names have been changed, make sure to propagate throughout the agent and the plugin files.

updated class names to StarcamHelper and StarcamAgent

sanahabhimani · 2023-05-25T17:28:21Z

socs/agents/starcam/agent.py

+        filter_return_image_bool = 0
+        n_sigma_value = 2
+        star_spacing_value = 15
+        self.cmds_for_camera = struct.pack('ddddddfiiiiiiiiiifffffffff', logodds, latitude, longitude, height, exposure, timelimit, set_focus_to_amount, auto_focus_bool, start_focus, end_focus, step_size, photos_per_focus, infinity_focus_bool, set_aperture_steps, max_aperture_bool, make_HP_bool, use_HP_bool, spike_limit_value, dynamic_hot_pixels_bool, r_smooth_value, high_pass_filter_bool, r_high_pass_filter_value, centroid_search_border_value, filter_return_image_bool, n_sigma_value, star_spacing_value)


I see that the output for this method is a long list of important values; we probably can't get around having that long list but I think there is a way to clean up the output in list format so that it's not one long line.

It's also good to set a new variable name for that list that you return at the end of this function instead of self.cmds_for_camera.

added all values to a list (named values)

packed and sent said commands

added a return values

this means i combined pack_cmds and send_cmds into a single function, now named pack_and_send_cmds. propagated this change to their call in send_commands

values = [logodds, latitude, longitude, height, exposure, timelimit, set_focus_to_amount, auto_focus_bool, start_focus, end_focus, step_size, photos_per_focus, infinity_focus_bool, set_aperture_steps, max_aperture_bool, make_HP_bool, use_HP_bool, spike_limit_value, dynamic_hot_pixels_bool, r_smooth_value, high_pass_filter_bool, r_high_pass_filter_value, centroid_search_border_value, filter_return_image_bool, n_sigma_value, star_spacing_value] # pack values into the command for the camera self.cmds_for_camera = struct.pack('ddddddfiiiiiiiiiifffffffff', *values) # send commands to the camera self.comm.sendto(self.cmds_for_camera, (self.ip, self.port)) print(“Commands sent to camera”) # Return the list of values return values

sanahabhimani · 2023-05-25T17:39:50Z

socs/agents/starcam/agent.py

+        c_time = starcamdata_unpacked[0]
+        gmt = starcamdata_unpacked[1]
+        blob_num = starcamdata_unpacked[2]
+        obs_ra = starcamdata_unpacked[3]
+        astrom_ra = starcamdata_unpacked[4]
+        obs_dec = starcamdata_unpacked[5]
+        astrom_dec = starcamdata_unpacked[6]
+        fr = starcamdata_unpacked[7]
+        ps = starcamdata_unpacked[8]
+        alt = starcamdata_unpacked[9]
+        az = starcamdata_unpacked[10]
+        ir = starcamdata_unpacked[11]
+        astrom_solve_time = starcamdata_unpacked[12]
+        camera_time = starcamdata_unpacked[13]
+        return c_time, gmt, blob_num, obs_ra, astrom_ra, obs_dec, astrom_dec, fr, ps, alt, az, ir, astrom_solve_time, camera_time


Can use a similar approach as here: simonsobs/soaculib@ebd2a9d

This will clean up the astrometry data a bit and allow you to return a dictionary of that data

That example would probably be useful when using struct.pack() in pack_cmds(). But +1 for returning a dict from this method, that'll especially clean up the call to this method on line 146. (And potentially lines 147-160 if you just use the same keys as you want to insert into the data dict.)

used the above approach as a guide. now in get_astrom_data() a list of keys is made, then a dictionary is created from the unpacked data and is returned.

def get_astrom_data(self): """Receive data from camera and unpack it""" (starcamdata_raw, _) = self.comm.recvfrom(224) starcamdata_unpacked = struct.unpack_from("dddddddddddddiiiiiiiiddiiiiiiiiiiiiiifiii", starcamdata_raw) starcam_data_keys = ['c_time', 'gmt', 'blob_num', 'obs_ra', 'astrom_ra', 'obs_dec', 'fr', 'ps', 'alt', 'az', 'ir', 'astrom_solve_time', 'camera_time'] # Create a dictionary of the unpacked data astrom_data = [starcamdata_unpacked[i] for i in range(len(starcam_data_keys))] astrom_data_dict = {keys[i]: astrom_data[i] for i in range(len(starcam_data_keys ))} return astrom_data_dict

this does change what was on lines 146-160 quite a bit. Now looks like this

# get astrometry data astrom_data = self.StarcamHelper.get_astrom_data() # update the data dictionary, update the session, and publish data['data'].update(astrom_data_dict) session.data.update(data['data']) self.agent.publish_to_feed('starcamera', data)

sanahabhimani · 2023-05-25T17:40:39Z

socs/agents/starcam/agent.py

+        self.comm.close()
+
+
+class starcam_Agent:


Apart from changing the style of this class name to pep8, it's also useful to name it using the company that the agent device is from. Especially because we will have a different star camera agent for the SATs.

You'll similarly want to change the directory name from starcam to something more specific

changed the directory from starcam to starcam_lat.

sanahabhimani · 2023-05-25T17:42:37Z

socs/agents/starcam/agent.py

+    def send_cmds(self):
+        self.comm.sendto(self.cmds_for_camera, (self.ip, self.port))
+        print("Commands sent to camera")


Combine send_cmds and pack_cmds into one function since send_cmds is expecting the output from pack_cmds

sanahabhimani · 2023-05-26T17:50:42Z

socs/agents/starcam/agent.py

+            params = {}
+        with self.lock.acquire_timeout(timeout=100, job='init') as acquired:
+            if not acquired:
+                self.log.warn("Could not start init because {} is already running".format(self.lock.job))


Also need to change init in this message to acq

sanahabhimani · 2023-05-26T17:52:22Z

socs/agents/starcam/agent.py

+        from argparse import ArgumentParser as A
+        parser_in = A()


Since A() is never used, remove as A in line 177 and remove line 178

Line 178 is actually still needed, parser_in is used below.

But along these lines, move the argparse import to the top of the file as just import argparse and make these lines:

if parser is None: parser = argparse.ArgumentParser()

for consistency with other agents. (Also make the argument parser instead of parser_in.)

sanahabhimani · 2023-05-26T17:53:26Z

socs/agents/starcam/agent.py

+def main(args=None):
+    parser = add_agent_args()
+    args = site_config.parse_args(agent_class="starcam_Agent", parser=parser)
+    startup = True


Remove startup=True, as it's already used a couple lines below when registering a task

sanahabhimani · 2023-05-26T19:07:13Z

socs/agents/starcam/agent.py

+        self.comm.settimeout(timeout)
+
+    def pack_cmds(self):
+        """pack commands and parameters to be sent to star camera"""


Change doc strings to be consistent with the rest of the script; there are other functions where the doc strings should also be changed but not commenting there for the sake of redundancy.

sanahabhimani · 2023-05-26T19:52:37Z

socs/agents/starcam/agent.py

+
+        Parameters:
+            test_mode (bool, optional): Run the acq process loop only once. This is meant only for testing. Default is false.
+        """


Also need to add session.data output in your acq docstring

BrianJKoopman

Yes, thanks @karenperezsarmiento! And thanks @sanahabhimani for the thorough review!

I've commented on some points that Sanah brought up, and also have a structural question about the way the task and process interact to command the starcamera.

BrianJKoopman · 2023-06-08T18:00:46Z

socs/agents/starcam/agent.py

+
+    def send_commands(self, session, params=None):


The parts needed for an initialization task are in lines 101-104, where the starcam_Helper class (the class that sets up the socket connection to the hardware) is initiated.

BrianJKoopman · 2023-06-08T18:03:13Z

socs/agents/starcam/agent.py

+        from argparse import ArgumentParser as A
+        parser_in = A()


Line 178 is actually still needed, parser_in is used below.

But along these lines, move the argparse import to the top of the file as just import argparse and make these lines:

if parser is None: parser = argparse.ArgumentParser()

for consistency with other agents. (Also make the argument parser instead of parser_in.)

BrianJKoopman · 2023-06-08T18:14:47Z

socs/agents/starcam/agent.py

+        c_time = starcamdata_unpacked[0]
+        gmt = starcamdata_unpacked[1]
+        blob_num = starcamdata_unpacked[2]
+        obs_ra = starcamdata_unpacked[3]
+        astrom_ra = starcamdata_unpacked[4]
+        obs_dec = starcamdata_unpacked[5]
+        astrom_dec = starcamdata_unpacked[6]
+        fr = starcamdata_unpacked[7]
+        ps = starcamdata_unpacked[8]
+        alt = starcamdata_unpacked[9]
+        az = starcamdata_unpacked[10]
+        ir = starcamdata_unpacked[11]
+        astrom_solve_time = starcamdata_unpacked[12]
+        camera_time = starcamdata_unpacked[13]
+        return c_time, gmt, blob_num, obs_ra, astrom_ra, obs_dec, astrom_dec, fr, ps, alt, az, ir, astrom_solve_time, camera_time


That example would probably be useful when using struct.pack() in pack_cmds(). But +1 for returning a dict from this method, that'll especially clean up the call to this method on line 146. (And potentially lines 147-160 if you just use the same keys as you want to insert into the data dict.)

BrianJKoopman · 2023-06-08T18:24:09Z

socs/agents/starcam/agent.py

I'd like to confirm the usage of this agent, as the task/process structure seems a bit unusual to me. The send_commands() task runs when you call it in a client to actually send the commands to the star camera, and the acq() process runs continuously to receive the response from the starcamera.

Do you keep acq() running and only call send_commands() when you need to take a set of photos with the camera? Is there always a response to be receiving in get_astrom_data? What happens if you start acq() but never call send_commands()?

What I would have expected was either:

The acq() process sent commands each time before trying to receive a response with get_astrom_data(), and then you likely don't need a separate task, unless running a manual trigger is useful.

The send_commands() task also received the response after sending commands. This would be just a manual version of the acq() process essentially.

I'm not sure which of these two makes more sense, as I'm not familiar with the actual use of the camera. But the current implementation seems like sort of a hybrid since commands are sent by the task and the response received in the process.

thanks so much @karenperezsarmiento, @sanahabhimani, @BrianJKoopman for all your thorough work on this! going to try addressing all these comments in the coming days since the starcam will start becoming super useful in just another few months:)

to address your last comment, brian, your first paragraph above is spot on. i believe send_commands() is called upon startup to set various camera+astrometry parameters. these params are fed to the starcam computer and set things like exposure time (and other camera params like autofocusing), and lat/long (which affect its ability to find an Az El pointing solution). the camera software should be running a program that takes these parameters and is continuously taking pictures, finding pointing solutions, and delivering packets of info to the agent via get_astrom_data() (at a cadence determined by the exposure time + the time it takes for a pointing solution to be found in software). if send_commands() were to not be called, there should be defaults set locally for each of these parameters in this starcam software that would be used in their stead.

thanks again!

…ocs/socs/agents/starcam_lat in agent.py: - changed class names from starcam_Helper and starcam_Agent to StarcamHelper and StarcamAgent - combined pack_cmds() and send_cmds() into single function: pack_and_send_cmds() - in pack_and_send_cmds(), added all values to a list (named values) packed and sent said commands added a return values - in get_astrom_data(), added a list of keys for a dictionary made dictionary from unpacked data added a return for this dictionary - in acq(), replace dictionary definition given the changes to get_astrom_data() changed job = 'init' to job = 'acq' - in add_agent_args() removed defult ip address changed --user-port to --port change all instances of parser_in to parser moved import statement to top of file - in main() removed startup=True added txaio commands for logging and import txaio at top of file - changed doc strings across file - insterted ocs param decorator above send_commands() - changed latitude, longitude, and height params in send_commands() to reflect chilean coords

for more information, see https://pre-commit.ci

- reduced line lengths to satisfy pep8 rules -- some variable names were adjusted to accomodate this change

…tarcamera changes: - reduced line lengths to satisfy pep8 rules -- some variable names changed to accomodate this change

for more information, see https://pre-commit.ci

- changed invalid character in print statement to pass flake8 check

BrianJKoopman

Hi @AlexManduca, sorry I haven't had a chance to review your recent changes in detail yet, but there are a couple of things to address first anyway. In addition to the comment below the pre-commit checks are failing due to invalid characters in the code, see details here: https://results.pre-commit.ci/run/github/186511668/1732120479.uB-orrk5RL2nfbGl2oGNXg

(This link is linked at the bottom of the PR if you click "details" next to "pre-commit.ci - pr" in the checks section.)

BrianJKoopman · 2024-11-20T19:03:55Z

socs/agents/starcam_lat/agent.py

+<<<<<<< HEAD
+                # update the data dictionary+session and publish
+                data['data'].update(astrom_data_dict)
+=======
+                # update the data dictionary, update the session, and publish
+                data['data'].update(astrom_data_dict)
+>>>>>>> 82252e753327de73c687591486c64dca8f4a8638


This is indicative of a merge that wasn't handled properly. When there's a merge conflict the conflicting blocks are presented side by side like this and must be merged manually. Please resolve the three places this happened in this commit: 04999b4

will resolve and push again - thanks for the checkup

…tarcamera

- fixed invalid character to pass pre-commit check - resolved merge conflict - changed a couple variable names that were wrong before

- added import for os environ

for more information, see https://pre-commit.ci

- corrected typo in import - deleted 'LOG =' for txaio logging in main since LOG is never used

BrianJKoopman self-requested a review August 2, 2022 18:23

BrianJKoopman assigned karenperezsarmiento Aug 2, 2022

BrianJKoopman added the new agent New OCS agent needs to be created label Aug 2, 2022

BrianJKoopman requested a review from sanahabhimani May 22, 2023 15:42

sanahabhimani changed the base branch from develop to main May 25, 2023 17:30

sanahabhimani requested changes May 26, 2023

View reviewed changes

BrianJKoopman reviewed Jun 8, 2023

View reviewed changes

BrianJKoopman assigned AlexManduca and unassigned karenperezsarmiento Aug 27, 2024

AlexManduca force-pushed the starcamera branch from 001a752 to 3fc8d8b Compare November 13, 2024 19:23

pre-commit-ci bot and others added 5 commits November 13, 2024 19:23

[pre-commit.ci] auto fixes from pre-commit.com hooks

82252e7

for more information, see https://pre-commit.ci

changes:

f7a0565

- reduced line lengths to satisfy pep8 rules -- some variable names were adjusted to accomodate this change

Merge branch 'starcamera' of https://github.com/simonsobs/socs into s…

04999b4

…tarcamera changes: - reduced line lengths to satisfy pep8 rules -- some variable names changed to accomodate this change

[pre-commit.ci] auto fixes from pre-commit.com hooks

735850d

for more information, see https://pre-commit.ci

changes

0733d7a

- changed invalid character in print statement to pass flake8 check

BrianJKoopman requested changes Nov 20, 2024

View reviewed changes

tanaybhandarkar and others added 5 commits November 20, 2024 16:07

Merge branch 'starcamera' of https://github.com/simonsobs/socs into s…

3ea0cce

…tarcamera

changes:

6465c4f

- fixed invalid character to pass pre-commit check - resolved merge conflict - changed a couple variable names that were wrong before

changes:

10db58a

- added import for os environ

[pre-commit.ci] auto fixes from pre-commit.com hooks

824305f

for more information, see https://pre-commit.ci

changes:

d2ca913

- corrected typo in import - deleted 'LOG =' for txaio logging in main since LOG is never used

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Starcamera - first PR request for testing starcamera at the LAT #335

Starcamera - first PR request for testing starcamera at the LAT #335

karenperezsarmiento commented Aug 1, 2022 •

edited by BrianJKoopman

Loading

sanahabhimani left a comment

sanahabhimani May 25, 2023

AlexManduca Nov 13, 2024

sanahabhimani May 25, 2023

AlexManduca Nov 13, 2024

sanahabhimani May 25, 2023

BrianJKoopman Jun 8, 2023

AlexManduca Nov 13, 2024

sanahabhimani May 25, 2023

AlexManduca Nov 13, 2024

sanahabhimani May 25, 2023 •

edited

Loading

sanahabhimani May 26, 2023

sanahabhimani May 26, 2023

BrianJKoopman Jun 8, 2023

sanahabhimani May 26, 2023

sanahabhimani May 26, 2023

sanahabhimani May 26, 2023

BrianJKoopman left a comment

BrianJKoopman Jun 8, 2023

BrianJKoopman Jun 8, 2023

BrianJKoopman Jun 8, 2023

BrianJKoopman Jun 8, 2023

AlexManduca Nov 12, 2024

BrianJKoopman left a comment

BrianJKoopman Nov 20, 2024

AlexManduca Nov 20, 2024

		from ocs.ocs_twisted import TimeoutLock


		class starcam_Helper:

Starcamera - first PR request for testing starcamera at the LAT #335

Are you sure you want to change the base?

Starcamera - first PR request for testing starcamera at the LAT #335

Conversation

karenperezsarmiento commented Aug 1, 2022 • edited by BrianJKoopman Loading

Description

Motivation and Context

How Has This Been Tested?

Types of changes

Checklist:

sanahabhimani left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanahabhimani May 25, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BrianJKoopman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BrianJKoopman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karenperezsarmiento commented Aug 1, 2022 •

edited by BrianJKoopman

Loading

sanahabhimani May 25, 2023 •

edited

Loading