selfcal-make_sourcedb_target error: 'endPtr == str+end; false is not a floating point value' #32

mknapp55 · 2017-08-01T23:32:19Z

2017-08-01 23:26:31 DEBUG   node.ip-100-64-109-221.executable_args: infile            = /data/t1data/lba/working_L525878/pipeline/L525880_SB000_uv.selfcal-filter_model
2017-08-01 23:26:31 DEBUG   node.ip-100-64-109-221.executable_args: executable        = /usr/bin/makesourcedb
2017-08-01 23:26:31 DEBUG   node.ip-100-64-109-221.executable_args: working directory = /data/t1data/lba/working_L525878/pipeline
2017-08-01 23:26:31 DEBUG   node.ip-100-64-109-221.executable_args: arguments         = []
2017-08-01 23:26:31 DEBUG   node.ip-100-64-109-221.executable_args: arg dictionary    = {'format': '<', 'outtype': 'blob', 'in': '/data/t1data/lba/working_L525878/pipeline/L525880_SB000_uv.selfcal-filter_
model', 'out': '/data/t1data/lba/working_L525878/pipeline/L525880_SB000_uv.selfcal-make_sourcedb_target'}
2017-08-01 23:26:31 DEBUG   node.ip-100-64-109-221.executable_args: environment       = {'OMP_NUM_THREADS': '8', 'PATH': '/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin', 'LOFARROOT': '/usr', 'PYTHONPATH': '/usr/lib/python2.7/dist-packages:/data/scripts/'}
2017-08-01 23:26:31 INFO    node.ip-100-64-109-221.executable_args: Processing /data/t1data/lba/working_L525878/pipeline/L525880_SB000_uv.selfcal-filter_model
2017-08-01 23:26:31 DEBUG   node.ip-100-64-109-221.executable_args.L525880_SB000_uv.selfcal-filter_model: Running: /usr/bin/makesourcedb format=< outtype=blob in=/data/t1data/lba/working_L525878/pipeline/L525880_SB000_uv.selfcal-filter_model out=/data/t1data/lba/working_L525878/pipeline/L525880_SB000_uv.selfcal-make_sourcedb_target
[...]
2017-08-01 23:26:31 DEBUG   node.ip-100-64-109-221.executable_args.L525880_SB000_uv.selfcal-filter_model: /usr/bin/makesourcedb stdout: /usr/bin/makesourcedb: Version GvD 2013-May-16
DEBUG - Assertion: endPtr == str+end; false is not a floating point value

2017-08-01 23:26:31 WARNING node.ip-100-64-109-221.executable_args.L525880_SB000_uv.selfcal-filter_model: /usr/bin/makesourcedb stderr: log4cplus:WARN Property configuration file "makesourcedb.log_prop" not found.
log4cplus:WARN Using basic logging configuration.
Caught LOFAR exception: [AssertError: Assertion: endPtr == str+end; false is not a floating point value]
in function double LOFAR::strToDouble(const string&)
(/build/lofar-b_IP79/lofar-2.20.2/LCS/Common/src/StringUtil.cc:446)

The text was updated successfully, but these errors were encountered:

adrabent · 2017-08-11T07:51:11Z

How does your skymodel

/data/t1data/lba/working_L525878/pipeline/L525880_SB000_uv.selfcal-filter_model

look like?

mknapp55 · 2017-08-11T13:34:16Z

Here's the top of the file:

ubuntu@ip-100-64-109-221:/data/t1data/lba/working_L525878/pipeline$ more L525880_SB000_uv.selfcal-filter_model
FORMAT = Name, Type, Ra, Dec, I, SpectralIndex='[]', LogarithmicSI, ReferenceFrequency='56834411.6211', MajorAxis, MinorAxis, Orientation

# LSMTool history:
# 2017-08-01 23:26:25: LOAD (from file '/data/t1data/lba/working_L525878/pipeline/L525880_SB000_uv_128888861t_77MHz_TC2.selfcal-image_high2-sources.txt')
# 2017-08-01 23:26:26: FILTER (/data/t1data/lba/working_L525878/pipeline/L525880_SB000_uv_128888861t_77MHz_TC2.selfcal-image_high1-MFS-image.selfcal-mask_high = True None)
# 2017-08-01 23:26:26: FILTER (with array of indices/bools)

s0c46, POINT, -0:56:32.921, -8.54.33.213, -0.0167820456667, [0.127654506271412], false, 56834411.6211
s0c47, POINT, -0:56:34.54, -8.54.33.185, 0.00325622398287, [-0.0234185208168157], false, 56834411.6211
s0c48, POINT, -0:56:35.35, -8.54.33.171, 0.00890620053695, [-0.0554472863117529], false, 56834411.6211
s0c49, POINT, -0:56:34.539, -8.54.21.157, 0.0252640648268, [-0.0765094483542936], false, 56834411.6211
s0c50, POINT, -0:56:30.488, -8.54.09.199, 0.0362661344977, [-0.1762711316591], false, 56834411.6211
s0c51, POINT, -0:56:32.107, -8.54.09.172, 0.0506629916838, [-0.10862780049329], false, 56834411.6211
s0c52, POINT, -0:56:33.727, -8.54.09.144, 0.00139489346574, [0.00103313074374939], false, 56834411.6211
s0c53, POINT, -0:56:32.106, -8.53.57.144, 0.05588815352, [-0.151045000780757], false, 56834411.6211
s0c54, POINT, -0:56:32.916, -8.53.57.13, 0.00268540060061, [0.0149230287659617], false, 56834411.6211
s0c55, POINT, -0:56:32.104, -8.53.45.116, 0.0463246607683, [-0.136477504184168], false, 56834411.6211
s0c56, POINT, -0:56:35.343, -8.53.45.06, 0.0176961035688, [-0.0239132830105105], false, 56834411.6211
s0c57, POINT, -0:56:32.102, -8.53.33.089, 0.0309555218621, [-0.0968894716063006], false, 56834411.6211
s0c59, POINT, -0:56:32.101, -8.53.21.061, 0.0709517756271, [-0.187908560922995], false, 56834411.6211
s0c60, POINT, -0:56:32.911, -8.53.21.048, 0.03492492324, [-0.0683533725485374], false, 56834411.6211
s0c61, POINT, -0:56:34.53, -8.53.21.02, 0.0124026410058, [0.0122583669019506], false, 56834411.6211

It has ~5600 sources. I can upload the whole file if that would be helpful - just let me know.

ajstewart · 2017-08-17T10:58:08Z

I ran into this a while ago - the 'LogarithmicSI' value is a sky model feature that is readable in the newer version of makesourcedb, that (at least to my knowledge) has not been incorporated into the lofar software release yet.

But the feature is already in the LSMtool release.

Francesco had been testing newer versions of NDPPP and makesourcedb in his dd pipeline which is how I ran across it back in May.

In his scripts there is 'ft_model_wsclean' and 'ft_model_cc', and I think PiLL is using the sky model filter method from the latter function, which in turn uses LSMtool. The 'wsclean' version uses PyBDSM/F to create a mask from the image as oppose to using the clean components from wsclean, hence LSMtool is not used.

So I believe you could either downgrade LSMtool (I think the change came in around April) or create a toggle and add the other method in the PiLL script filter_model.py while the newer version of makesourcedb is not available.

EDIT: I meant to say that downgrading LSMtool may create other issues so getting the other method in there might be the safer option for now.

adrabent · 2017-08-18T08:34:31Z

I see.
I am using lsmtool 1.1.0 and it works fine as far as I can judge.
The issue with the other procedure is that Francesco uses an executable called nnradd in his function ft_model_wsclean. Is this a public tool? I might ask Francesco about it.

ajstewart · 2017-08-18T08:51:47Z

Ah yes that nnradd was written by Andre to resample images to a different pixel size which I obtained through Francesco and is a standalone install package (i.e. not a python script). As this is not easily available and will soon be redundant anyway, perhaps requiring LSMtool <= 1.1 for PiLL is the current solution until the new makesourcedb is ready?

revoltek · 2017-08-18T08:54:55Z

for the DD cal pipeline LSMtool 1.2 is required, maybe also for the selfcal part (but only if you use the clean component predict)

mknapp55 · 2017-09-14T21:42:20Z

I'm using the most up-to-date LSMtool from github. I think the makesourcedb executable is from the LOFAR bundle (2.20 I think), however. Do you know which version of the LOFAR software has an appropriately up-to-date makesourcedb? Or does that not exist yet? As for downgrading LSMtool to 1.1....I'm not actually sure how to do that?? Is that really the best/only way around this error?

ajstewart · 2017-09-14T21:56:10Z

The newer makesourcedb is not released yet (again, as far as I'm aware).

How did you install LSMtool into your environment? For example I always use pip to install so it's just a case of pip uninstall. If you did python setup.py install then I think you may have to simply delete the files manually. Though I think perhaps pip is clever and can work it out if you just do a pip uninstall. Then reinstall the 1.1 version from GitHub.

Though I see 1.2 is the first 'release' on GitHub, the commit you want to avoid is db2c16c on the 22 April. Try pulling the code after the Feb 14 commit, there seems to be a long gap there.

Edit: pip list will show you what's installed, and you should see lsmtool. So pip uninstall lsmtool should work fine.

mknapp55 · 2017-09-14T22:07:51Z

I installed it using the setup.py. I can roll back to Feb 14 as you suggest and install from there. Have you encountered any issues in selfcal from skipping back to an older version?

ajstewart · 2017-09-14T22:13:06Z

I haven't personally rolled back yet as I was using Francesco's scripts and just switched to the ft_model_wsclean function. But as far as I'm aware you shouldn't run into any issues, nothing new is used, so just getting rid of that LogarithmicSI column so that makesourcedb can build the model correctly should do the trick.

Edit: Plus Alexander reports that it runs ok for him with 1.1.

mknapp55 · 2017-09-14T22:35:46Z

I rolled back the LSMTool version (lsmtool --version gives me 1.1.0), but I'm still getting the same error. Do I need to roll the pipeline back to a certain step for this to work? If so, how far back do I need to go?

ajstewart · 2017-09-14T22:55:03Z

Looking at it I believe you need to roll the pipeline back one step (assuming this is Step 15 in the selfcal.parset, the first case of makesourcedb).

It needs to run Step 14b again - the filter_model step. That's the step that generates the model that makesourcedb uses, in which the script uses lsmtool.

mknapp55 · 2017-09-18T21:35:05Z

I tried that, and then I tried removing the statefile entirely when the filter_model rollback didn't work. My parset only specifies the selfcal pipeline step because the other two steps are already done. I still get this error:
ERROR: Column 'logarithmicsi' is not currently allowed
Where is that column? Do I need to rerun time_split to re-generate MSs without the offending column?

ajstewart · 2017-09-19T11:47:36Z

Ah ok, maybe it comes earlier then. In your original error at the beginning it is the filter model that it trips up on:

2017-08-01 23:26:31 DEBUG   node.ip-100-64-109-221.executable_args.L525880_SB000_uv.selfcal-filter_model: /usr/bin/makesourcedb stdout: /usr/bin/makesourcedb: Version GvD 2013-May-16
DEBUG - Assertion: endPtr == str+end; false is not a floating point value

This LogarithmicSI column is in a sky model that is being attempted to be run through makesourcedb, the sky model will look like that below. But this column is only there in sky models that are created using the LSMtool (after the April update). I haven't run a new PiLL run yet so I can't see for myself, perhaps @adrabent can confirm exactly what sky model is attempted to be used. It should just be a case of replacing that sky model with the column removed (i.e. running filter model again with the original model with the old LSMtool).

Format = Name, Type, Ra, Dec, I, SpectralIndex, LogarithmicSI, ReferenceFrequency='66697692.8710938', MajorAxis, MinorAxis, Orientation
s0c0,POINT,-00:31:47.311,82.33.59.968,0.000821110153790913,[0.00899086983409225],false,66697692.8710938,,,
s0c1,POINT,-00:31:53.007,82.33.55.257,0.00633233500220994,[0.0448726827385555],false,66697692.8710938,,,
s0c2,POINT,-00:32:04.392,82.33.45.822,0.000384514107959337,[0.0147373734728456],false,66697692.8710938,,,
s0c3,POINT,-00:32:15.769,82.33.36.369,0.00298350900071676,[0.012113355037623],false,66697692.8710938,,,
s0c4,POINT,-00:32:21.454,82.33.31.635,0.00211496110134362,[0.0172073610519028],false,66697692.8710938,,,
s0c5,POINT,-00:32:32.819,82.33.22.153,0.00651739500370666,[0.0531698135529002],false,66697692.8710938,,,
s0c6,POINT,-00:32:38.499,82.33.17.406,0.0104992299303997,[0.0687451027864146],false,66697692.8710938,,,
s0c7,POINT,-00:32:44.176,82.33.12.654,0.00981892578851219,[0.0399825889206263],false,66697692.8710938,,,
s0c8,POINT,-00:32:49.852,82.33.07.897,0.00641159026375492,[0.00802698163456774],false,66697692.8710938,,,
...

ajstewart · 2017-09-19T14:17:52Z

@mknapp55 @adrabent

Ok I've looked deeper into this as something didn't seem right and I've got the origin wrong:

The LogarithmicSI sky model column is introduced by WSClean not LSMtool. It's the -save-source-list option (see here https://sourceforge.net/p/wsclean/wiki/ComponentList/)
This is why I switched to the other method as it did not depend on the wsclean list at all. It used pybdsf to extract a sky model instead.
This means that LSMtool version 1.2 is fine to use with PiLL.
The (temporary) solution without using the other method is to remove the LogarithmicSI column from the model before or after the filter as been applied, such that makesourcedb can build it.

I'm very sorry for the confusion!

ajstewart · 2017-09-20T10:53:57Z

@mknapp55 - I've written a potential fix for this by including an option to remove the LogarithmicSI column in the filter_model.py script that is called. See the branch here:

https://github.com/ajstewart/PiLL/tree/temp-fix-for-LogarithmicSI

Note that only that script and the selfcal.parset is changed, with the latter only adding three lines to turn on the option.

As you have a processed run up to that point, you could firstly test the filter_model.py script manually by copying the sky model generated from wsclean along with the mask and the ms file somewhere and running just the script to see if it generates the model as expected (I believe you're looking for *image_high2-sources.txt and the mask_high output.)

If it works then if you could test it in the pipeline environment that would be great, the first filter_model is the step to go from. It will take me a few days to catch up with a pipeline to test it myself.

mknapp55 · 2017-09-22T18:43:07Z

@ajstewart I'm working with your modified filter_model.py now. There was a minor bug related to the naming of the argument to delete the bad column. Once I fixed that, I ran into the error below. I'm working on tracking it down, but I though I'd post it in case you have a quick fix.

ubuntu@ip-100-64-109-221:/data/t1data/lba/working_L525878/pipeline$ python /data/PiLL/scripts/filter_model.py --ms /data/t1data/lba/L525880/L525880_SB000_uv.MS --output /data/t1data/lba/working_L525878/pipeline/L525880_SB000_uv.selfcal-filter_model -m /data/t1data/lba/working_L525878/pipeline/L525880_SB000_uv_128888861t_77MHz_TC2.selfcal-image_high1-MFS-image.selfcal-mask_high -s /data/t1data/lba/working_L525878/pipeline/L525880_SB000_uv_128888861t_77MHz_TC2.selfcal-image_high2-sources.txt -S /data/PiLL/scripts -x
/data/PiLL/scripts/filter_model.py:67: SyntaxWarning: import * only allowed at module level
  def main(ms,skymodel,maskname,skymodel_cut,scripts,keep_in_beam=True,delete_LogSI_column=False):
Traceback (most recent call last):
  File "/data/PiLL/scripts/filter_model.py", line 116, in <module>
    main(ms=args.ms,skymodel=args.skymodel,maskname=args.mask,skymodel_cut=args.output,scripts=args.scripts, delete_LogSI_column=args.removeLogSIcol) #delete LogSI required for old makesourcedb.
  File "/data/PiLL/scripts/filter_model.py", line 70, in main
    from lib_pipeline import *
  File "/data/PiLL/scripts/lib_pipeline.py", line 345, in <module>
    from lib_pipeline_dd import *
  File "/data/PiLL/scripts/lib_pipeline_dd.py", line 3, in <module>
    from astropy.table import Table
  File "/usr/lib/python2.7/dist-packages/astropy/__init__.py", line 112, in <module>
    _check_numpy()
  File "/usr/lib/python2.7/dist-packages/astropy/__init__.py", line 100, in _check_numpy
    from .utils import minversion
  File "/usr/lib/python2.7/dist-packages/astropy/utils/__init__.py", line 18, in <module>
    from .misc import *
  File "/usr/lib/python2.7/dist-packages/astropy/utils/misc.py", line 28, in <module>
    from ..utils.compat.odict import OrderedDict
  File "/usr/lib/python2.7/dist-packages/astropy/utils/compat/__init__.py", line 14, in <module>
    from .numpycompat import *
  File "/usr/lib/python2.7/dist-packages/astropy/utils/compat/numpycompat.py", line 18, in <module>
    NUMPY_LT_1_6_1 = not minversion('numpy', '1.6.1')
  File "/usr/lib/python2.7/dist-packages/astropy/utils/introspection.py", line 154, in minversion
    from pkg_resources import parse_version
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2927, in <module>
    @_call_aside
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2913, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2940, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 626, in _build_master
    ws = cls()
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 619, in __init__
    self.add_entry(entry)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 673, in add_entry
    self.entry_keys.setdefault(entry, [])
TypeError: unhashable type: 'list'

ajstewart · 2017-09-22T19:13:46Z

Sorry for the minor bug I changed the name of the function last minute so I'm not surprised I missed something - I also see it now.

That looks like an import error on something not to do with my edits - possibly to do with the from lib_pipeline import * within the main function. Try moving these outside the main function just for when you're running it manually for now, see if that makes a difference?

Edit: By 'these' I mean

sys.path.append(scripts)
from lib_pipeline import *

to the normal import bit. If running in the same environment as PiLL works I'm not sure why the script running standalone wouldn't import properly.

mknapp55 · 2017-09-22T19:37:09Z

For reference, this is what I get in the pipeline framework using the script with the args bug fixed:

2017-09-22 19:34:37 INFO    genericpipeline.executable_args: Starting /lofsoft/PiLL/scripts/filter_model.py run
Reading configuration file: /data/t1data/lba/working_L525878/pipeline.cfg
Reading task definition file(s): /usr/share/pipeline/tasks.cfg
2017-09-22 19:34:37 INFO    genericpipeline.executable_args: Limiting to 2 simultaneous jobs/node
2017-09-22 19:34:37 INFO    genericpipeline.executable_args: ********************** Remote method is local
2017-09-22 19:34:37 INFO    genericpipeline.executable_args: Waiting for compute threads...
2017-09-22 19:34:37 INFO    genericpipeline.executable_args: Subprocess starting: /bin/sh -c python /usr/lib/python2.7/dist-packages/lofarpipe/recipes/nodes/python_plugin.py 0 127.0.1.1 44728 (['/bin/sh', '-c', 'python /usr/lib/python2.7/dist-packages/lofarpipe/recipes/nodes/python_plugin.py 0 127.0.1.1 44728'])
2017-09-22 19:34:37 WARNING genericpipeline.executable_args: /usr/lib/python2.7/dist-packages/lofarpipe/support/utilities.pyc : Using default subprocess module!
2017-09-22 19:34:37 WARNING genericpipeline.executable_args: /lofsoft/PiLL/scripts/filter_model.py:67: SyntaxWarning: import * only allowed at module level
2017-09-22 19:34:37 WARNING genericpipeline.executable_args:   def main(ms,skymodel,maskname,skymodel_cut,scripts,keep_in_beam=True,delete_LogSI_column=False):
2017-09-22 19:34:37 INFO    node.ip-100-64-109-221.python_plugin: Processing /data/t1data/lba/L525880/L525880_SB000_uv.MS
2017-09-22 19:34:39 WARNING genericpipeline.executable_args: WARNING: FITSFixedWarning: 'unitfix' made the change 'Changed units: 'HZ' -> 'Hz''. [astropy.wcs.wcs]
2017-09-22 19:34:39 WARNING genericpipeline.executable_args: WARNING: FITSFixedWarning: 'unitfix' made the change 'Changed units: 'HZ' -> 'Hz''.
2017-09-22 19:34:39 WARNING genericpipeline.executable_args: WARNING: AstropyDeprecationWarning: The ascard function is deprecated and may be removed in a future version.
2017-09-22 19:34:39 WARNING genericpipeline.executable_args:         Use the `.cards` attribute instead. [astropy.utils.decorators]
2017-09-22 19:34:39 WARNING genericpipeline.executable_args: WARNING: AstropyDeprecationWarning: The ascard function is deprecated and may be removed in a future version.
2017-09-22 19:34:39 WARNING genericpipeline.executable_args:         Use the `.cards` attribute instead.
2017-09-22 19:34:39 WARNING genericpipeline.executable_args: WARNING: AstropyDeprecationWarning: The CardList class has been deprecated; all its former functionality has been subsumed by the Header class, so CardList objects should not be directly created.  See the PyFITS 3.1.0 CHANGELOG for more details. [astropy.io.fits.card]
2017-09-22 19:34:39 WARNING genericpipeline.executable_args: WARNING: AstropyDeprecationWarning: The CardList class has been deprecated; all its former functionality has been subsumed by the Header class, so CardList objects should not be directly created.  See the PyFITS 3.1.0 CHANGELOG for more details.
2017-09-22 19:34:39 WARNING genericpipeline.executable_args: WARNING: AstropyDeprecationWarning: The ascard function is deprecated and may be removed in a future version.
2017-09-22 19:34:39 WARNING genericpipeline.executable_args:         Use the `.cards` attribute instead. [astropy.utils.decorators]
2017-09-22 19:34:39 WARNING genericpipeline.executable_args: WARNING: AstropyDeprecationWarning: The ascard function is deprecated and may be removed in a future version.
2017-09-22 19:34:39 WARNING genericpipeline.executable_args:         Use the `.cards` attribute instead.
2017-09-22 19:34:39 WARNING genericpipeline.executable_args: INFO: Predict (apply mask)...
2017-09-22 19:34:39 WARNING genericpipeline.executable_args: ERROR: Column 'logarithmicsi' is not currently allowed
2017-09-22 19:34:39 WARNING genericpipeline.executable_args: INFO: Total time 2.5512s; user time: 1.0160s; system time: 0.1920s
2017-09-22 19:34:39 WARNING genericpipeline.executable_args: DEBUG: Start time was 1506108877.1783s; end time was 1506108879.7297s
2017-09-22 19:34:39 ERROR   node.ip-100-64-109-221.python_plugin: Column 'logarithmicsi' is not currently allowed
2017-09-22 19:34:39 INFO    node.ip-100-64-109-221.python_plugin: Total time 2.5512s; user time: 1.0160s; system time: 0.1920s
2017-09-22 19:34:40 WARNING genericpipeline.executable_args: 
2017-09-22 19:34:40 INFO    genericpipeline.executable_args: Subprocess completed with exit status 1: /bin/sh -c python /usr/lib/python2.7/dist-packages/lofarpipe/recipes/nodes/python_plugin.py 0 127.0.1.1 44728
2017-09-22 19:34:40 ERROR   genericpipeline.executable_args: Remote process python /usr/lib/python2.7/dist-packages/lofarpipe/recipes/nodes/python_plugin.py ['/data/t1data/lba/L525880/L525880_SB000_uv.MS', '/lofsoft/PiLL/scripts/filter_model.py', [], {'maskname': '/data/t1data/lba/working_L525878/pipeline/L525880_SB000_uv_128888861t_77MHz_TC2.selfcal-image_high1-MFS-image.selfcal-mask_high', 'skymodel_cut': '/data/t1data/lba/working_L525878/pipeline/L525880_SB000_uv.selfcal-filter_model', 'delete_LogSI_column': 'True', 'ms': '/data/t1data/lba/L525880/L525880_SB000_uv.MS', 'scripts': '/lofsoft/PiLL/scripts', 'skymodel': '/data/t1data/lba/working_L525878/pipeline/L525880_SB000_uv_128888861t_77MHz_TC2.selfcal-image_high2-sources.txt'}, '/data/t1data/lba/working_L525878/pipeline', False, {'args_format_option_argument': '=', 'args_format_option': '-', 'args_format': 'gnu', 'args_formatlongoption': '--', 'args_format_argument': ''}, {'OMP_NUM_THREADS': '8', 'PATH': '/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin', 'LOFARROOT': '/usr', 'PYTHONPATH': '/usr/lib/python2.7/dist-packages:/data/scripts/'}] failed on localhost
2017-09-22 19:34:41 ERROR   genericpipeline.executable_args: A job has failed with returncode 1 and error_tolerance is not set. Bailing out!

ajstewart · 2017-09-22T19:42:26Z

Ok this:

2017-09-22 19:34:39 ERROR node.ip-100-64-109-221.python_plugin: Column 'logarithmicsi' is not currently allowed

is an error from LSMtool - which version are you using of that? Did you upgrade back to the latest version as that one should allow the column and avoid this error.

mknapp55 · 2017-09-22T19:50:48Z

Moving the import statement to the top and hardcoding the location of 'scripts' yields another lib_pipeline error (when running the script manually):

Traceback (most recent call last):
  File "/data/PiLL/scripts/filter_model.py", line 117, in <module>
    main(ms=args.ms,skymodel=args.skymodel,maskname=args.mask,skymodel_cut=args.output,scripts=args.scripts, delete_LogSI_column=args.removeLogSIcol) #delete LogSI required for old makesourcedb.
  File "/data/PiLL/scripts/filter_model.py", line 74, in main
    phasecentre = get_phase_centre(ms)
  File "/data/PiLL/scripts/lib_pipeline_ms.py", line 156, in get_phase_centre
    with tb.table(ms + '/FIELD', ack=False) as field_table:
TypeError: can only concatenate list (not "str") to list

In the pipeline framework, upgrading back to the most recent LSMTool yielded this:

2017-09-22 19:49:01 INFO    genericpipeline.executable_args: Starting /lofsoft/PiLL/scripts/filter_model.py run
Reading configuration file: /data/t1data/lba/working_L525878/pipeline.cfg
Reading task definition file(s): /usr/share/pipeline/tasks.cfg
2017-09-22 19:49:01 INFO    genericpipeline.executable_args: Limiting to 2 simultaneous jobs/node
2017-09-22 19:49:01 INFO    genericpipeline.executable_args: ********************** Remote method is local
2017-09-22 19:49:01 INFO    genericpipeline.executable_args: Waiting for compute threads...
2017-09-22 19:49:01 INFO    genericpipeline.executable_args: Subprocess starting: /bin/sh -c python /usr/lib/python2.7/dist-packages/lofarpipe/recipes/nodes/python_plugin.py 0 127.0.1.1 36127 (['/bin/sh', '-c', 'python /usr/lib/python2.7/dist-packages/lofarpipe/recipes/nodes/python_plugin.py 0 127.0.1.1 36127'])
2017-09-22 19:49:01 WARNING genericpipeline.executable_args: /usr/lib/python2.7/dist-packages/lofarpipe/support/utilities.pyc : Using default subprocess module!
2017-09-22 19:49:01 WARNING genericpipeline.executable_args: /lofsoft/PiLL/scripts/filter_model.py:67: SyntaxWarning: import * only allowed at module level
2017-09-22 19:49:01 WARNING genericpipeline.executable_args:   def main(ms,skymodel,maskname,skymodel_cut,scripts,keep_in_beam=True,delete_LogSI_column=False):
2017-09-22 19:49:01 INFO    node.ip-100-64-109-221.python_plugin: Processing /data/t1data/lba/L525880/L525880_SB000_uv.MS
2017-09-22 19:49:02 WARNING genericpipeline.executable_args: WARNING: FITSFixedWarning: 'unitfix' made the change 'Changed units: 'HZ' -> 'Hz''. [astropy.wcs.wcs]
2017-09-22 19:49:02 WARNING genericpipeline.executable_args: WARNING: FITSFixedWarning: 'unitfix' made the change 'Changed units: 'HZ' -> 'Hz''.
2017-09-22 19:49:02 WARNING genericpipeline.executable_args: WARNING: AstropyDeprecationWarning: The ascard function is deprecated and may be removed in a future version.
2017-09-22 19:49:02 WARNING genericpipeline.executable_args:         Use the `.cards` attribute instead. [astropy.utils.decorators]
2017-09-22 19:49:02 WARNING genericpipeline.executable_args: WARNING: AstropyDeprecationWarning: The ascard function is deprecated and may be removed in a future version.
2017-09-22 19:49:02 WARNING genericpipeline.executable_args:         Use the `.cards` attribute instead.
2017-09-22 19:49:02 WARNING genericpipeline.executable_args: WARNING: AstropyDeprecationWarning: The CardList class has been deprecated; all its former functionality has been subsumed by the Header class, so CardList objects should not be directly created.  See the PyFITS 3.1.0 CHANGELOG for more details. [astropy.io.fits.card]
2017-09-22 19:49:02 WARNING genericpipeline.executable_args: WARNING: AstropyDeprecationWarning: The CardList class has been deprecated; all its former functionality has been subsumed by the Header class, so CardList objects should not be directly created.  See the PyFITS 3.1.0 CHANGELOG for more details.
2017-09-22 19:49:02 WARNING genericpipeline.executable_args: WARNING: AstropyDeprecationWarning: The ascard function is deprecated and may be removed in a future version.
2017-09-22 19:49:02 WARNING genericpipeline.executable_args:         Use the `.cards` attribute instead. [astropy.utils.decorators]
2017-09-22 19:49:02 WARNING genericpipeline.executable_args: WARNING: AstropyDeprecationWarning: The ascard function is deprecated and may be removed in a future version.
2017-09-22 19:49:02 WARNING genericpipeline.executable_args:         Use the `.cards` attribute instead.
2017-09-22 19:49:02 WARNING genericpipeline.executable_args: INFO: Predict (apply mask)...
2017-09-22 19:49:03 WARNING genericpipeline.executable_args: WARNING: "/data/t1data/lba/working_l525878/pipeline/l525880_sb000_uv_128888861t_77mhz_tc2.selfcal-image_high1-mfs-image.selfcal-mask_high" is not a valid column. Trying it as a mask filename instead...
2017-09-22 19:49:04 WARNING genericpipeline.executable_args: INFO: Kept 3051 sources.
2017-09-22 19:49:04 WARNING genericpipeline.executable_args: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3293: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
2017-09-22 19:49:04 WARNING genericpipeline.executable_args: Check the NumPy 1.11 release notes for more information.
2017-09-22 19:49:04 WARNING genericpipeline.executable_args:   self.__setitem__(slice(i, j), value)
2017-09-22 19:49:05 WARNING genericpipeline.executable_args: INFO: Attempting to delete the 'LogarithmicSI' column from the sky model...
2017-09-22 19:49:05 WARNING genericpipeline.executable_args: ERROR: global name 'logger' is not defined
2017-09-22 19:49:05 WARNING genericpipeline.executable_args: INFO: Total time 4.5995s; user time: 4.1360s; system time: 0.2320s
2017-09-22 19:49:05 WARNING genericpipeline.executable_args: DEBUG: Start time was 1506109741.1950s; end time was 1506109745.7946s
2017-09-22 19:49:05 ERROR   node.ip-100-64-109-221.python_plugin: global name 'logger' is not defined
2017-09-22 19:49:05 INFO    node.ip-100-64-109-221.python_plugin: Total time 4.5995s; user time: 4.1360s; system time: 0.2320s
2017-09-22 19:49:06 WARNING genericpipeline.executable_args: 
2017-09-22 19:49:06 INFO    genericpipeline.executable_args: Subprocess completed with exit status 1: /bin/sh -c python /usr/lib/python2.7/dist-packages/lofarpipe/recipes/nodes/python_plugin.py 0 127.0.1.1 36127
2017-09-22 19:49:06 ERROR   genericpipeline.executable_args: Remote process python /usr/lib/python2.7/dist-packages/lofarpipe/recipes/nodes/python_plugin.py ['/data/t1data/lba/L525880/L525880_SB000_uv.MS', '/lofsoft/PiLL/scripts/filter_model.py', [], {'maskname': '/data/t1data/lba/working_L525878/pipeline/L525880_SB000_uv_128888861t_77MHz_TC2.selfcal-image_high1-MFS-image.selfcal-mask_high', 'skymodel_cut': '/data/t1data/lba/working_L525878/pipeline/L525880_SB000_uv.selfcal-filter_model', 'delete_LogSI_column': 'True', 'ms': '/data/t1data/lba/L525880/L525880_SB000_uv.MS', 'scripts': '/lofsoft/PiLL/scripts', 'skymodel': '/data/t1data/lba/working_L525878/pipeline/L525880_SB000_uv_128888861t_77MHz_TC2.selfcal-image_high2-sources.txt'}, '/data/t1data/lba/working_L525878/pipeline', False, {'args_format_option_argument': '=', 'args_format_option': '-', 'args_format': 'gnu', 'args_formatlongoption': '--', 'args_format_argument': ''}, {'OMP_NUM_THREADS': '8', 'PATH': '/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin', 'LOFARROOT': '/usr', 'PYTHONPATH': '/usr/lib/python2.7/dist-packages:/data/scripts/'}] failed on localhost
2017-09-22 19:49:07 ERROR   genericpipeline.executable_args: A job has failed with returncode 1 and error_tolerance is not set. Bailing out!

Progress!

ajstewart · 2017-09-22T20:16:01Z

Check the version I just pushed, just an edit to pass the logger from the main() to the delete_column(). Must be something in how the function is called within the generic pipeline. That should get past that one... I hope.

As for getting it to run manually that seems strange at the moment! I guess concentrate on the pipeline environment for now.

mknapp55 · 2017-09-22T20:25:17Z

Ok, that version runs through. It gives a lot of errors about strange sources, though...

2017-09-22 20:22:13 WARNING genericpipeline.executable_args: Check the NumPy 1.11 release notes for more information.
2017-09-22 20:22:13 WARNING genericpipeline.executable_args:   self.__setitem__(slice(i, j), value)
2017-09-22 20:22:15 WARNING genericpipeline.executable_args: INFO: Attempting to delete the 'LogarithmicSI' column from the sky model...
2017-09-22 20:22:15 WARNING genericpipeline.executable_args: ERROR: Source  is strange! Skipping!
2017-09-22 20:22:15 WARNING genericpipeline.executable_args: ERROR: Source # LSMTool history: is strange! Skipping!
2017-09-22 20:22:15 WARNING genericpipeline.executable_args: ERROR: Source # 2017-09-22 20:22:13: LOAD (from file '/data/t1data/lba/working_L525878/pipeline/L525880_SB000_uv_128888861t_77MHz_TC2.selfcal-image_high2-sources.txt') is strange! Skipping!
2017-09-22 20:22:15 WARNING genericpipeline.executable_args: ERROR: Source # 2017-09-22 20:22:13: FILTER (/data/t1data/lba/working_L525878/pipeline/L525880_SB000_uv_128888861t_77MHz_TC2.selfcal-image_high1-MFS-image.selfcal-mask_high = True None) is strange! Skipping!
2017-09-22 20:22:15 WARNING genericpipeline.executable_args: ERROR: Source  is strange! Skipping!
2017-09-22 20:22:15 WARNING genericpipeline.executable_args: ERROR: Source s0c59 is strange! Skipping!
[...]
2017-09-22 20:22:15 WARNING genericpipeline.executable_args: ERROR: Source s0c1796 is strange! Skipping!
2017-09-22 20:22:15 WARNING genericpipeline.executable_args: INFO: LogarithmicSI column successfully deleted.

ajstewart · 2017-09-22T20:32:10Z

Hmm interesting could you email me your sources list from WSClean please? I was using one that was produced from one of my images but maybe it's not up-to-date or representative of what could be in it. At least we know it can run the function now! Nearly there.

mknapp55 · 2017-09-22T20:39:18Z

I assume you mean the input skymodel (the .txt file)?

ajstewart · 2017-09-22T20:51:29Z

I assume you mean the input skymodel (the .txt file)?

Sorry, yes that's the one.

adrabent added a commit that referenced this issue Aug 18, 2017

requirements #32

793dba0

mknapp55 mentioned this issue Sep 24, 2017

Error at step selfcal-loop-steps #36

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

selfcal-make_sourcedb_target error: 'endPtr == str+end; false is not a floating point value' #32

selfcal-make_sourcedb_target error: 'endPtr == str+end; false is not a floating point value' #32

mknapp55 commented Aug 1, 2017

adrabent commented Aug 11, 2017

mknapp55 commented Aug 11, 2017

ajstewart commented Aug 17, 2017 •

edited

Loading

adrabent commented Aug 18, 2017

ajstewart commented Aug 18, 2017

revoltek commented Aug 18, 2017

mknapp55 commented Sep 14, 2017

ajstewart commented Sep 14, 2017 •

edited

Loading

mknapp55 commented Sep 14, 2017

ajstewart commented Sep 14, 2017 •

edited

Loading

mknapp55 commented Sep 14, 2017 •

edited

Loading

ajstewart commented Sep 14, 2017

mknapp55 commented Sep 18, 2017

ajstewart commented Sep 19, 2017

ajstewart commented Sep 19, 2017

ajstewart commented Sep 20, 2017

mknapp55 commented Sep 22, 2017

ajstewart commented Sep 22, 2017 •

edited

Loading

mknapp55 commented Sep 22, 2017

ajstewart commented Sep 22, 2017

mknapp55 commented Sep 22, 2017

ajstewart commented Sep 22, 2017

mknapp55 commented Sep 22, 2017

ajstewart commented Sep 22, 2017

mknapp55 commented Sep 22, 2017

ajstewart commented Sep 22, 2017

selfcal-make_sourcedb_target error: 'endPtr == str+end; false is not a floating point value' #32

selfcal-make_sourcedb_target error: 'endPtr == str+end; false is not a floating point value' #32

Comments

mknapp55 commented Aug 1, 2017

adrabent commented Aug 11, 2017

mknapp55 commented Aug 11, 2017

ajstewart commented Aug 17, 2017 • edited Loading

adrabent commented Aug 18, 2017

ajstewart commented Aug 18, 2017

revoltek commented Aug 18, 2017

mknapp55 commented Sep 14, 2017

ajstewart commented Sep 14, 2017 • edited Loading

mknapp55 commented Sep 14, 2017

ajstewart commented Sep 14, 2017 • edited Loading

mknapp55 commented Sep 14, 2017 • edited Loading

ajstewart commented Sep 14, 2017

mknapp55 commented Sep 18, 2017

ajstewart commented Sep 19, 2017

ajstewart commented Sep 19, 2017

ajstewart commented Sep 20, 2017

mknapp55 commented Sep 22, 2017

ajstewart commented Sep 22, 2017 • edited Loading

mknapp55 commented Sep 22, 2017

ajstewart commented Sep 22, 2017

mknapp55 commented Sep 22, 2017

ajstewart commented Sep 22, 2017

mknapp55 commented Sep 22, 2017

ajstewart commented Sep 22, 2017

mknapp55 commented Sep 22, 2017

ajstewart commented Sep 22, 2017

ajstewart commented Aug 17, 2017 •

edited

Loading

ajstewart commented Sep 14, 2017 •

edited

Loading

ajstewart commented Sep 14, 2017 •

edited

Loading

mknapp55 commented Sep 14, 2017 •

edited

Loading

ajstewart commented Sep 22, 2017 •

edited

Loading