Skip to content
This repository has been archived by the owner on Sep 24, 2021. It is now read-only.

让 list 输出对 pipe 更友好 #406

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Conversation

wlxiong
Copy link

@wlxiong wlxiong commented Aug 14, 2014

如果用户的离线文件多时, 一个习惯的动作是

$ lx list | head

但是现有实现会抛出如下异常:

IOError: [Errno 32] Broken pipe

这个修改尝试捕获这个异常.

此外, list 操作默认取回整个离线文件列表, 当文件数较多时速度会受网络延迟的影响, 所以修改为使用 generator 遍历文件列表, 这样就可以做到 lazy fetch 文件列表.

我的离线中有 600 多个文件, 修改前后执行 lx list | head 网络 IO 的对比如下:

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  162    3.029    0.019    3.029    0.019 {method 'recv' of '_socket.socket' objects}
    7    0.071    0.010    0.071    0.010 {method 'connect' of '_socket.socket' objects}

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   42    0.756    0.018    0.756    0.018 {method 'recv' of '_socket.socket' objects}
    1    0.014    0.014    0.014    0.014 {method 'connect' of '_socket.socket' objects}

@@ -103,6 +103,7 @@ def output_tasks(tasks, columns, args, top=True):
else:
raise NotImplementedError(k)
print
output.flush()
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

输出到 pipe 时, 每行输出一行都 flush 一次, 否则标准输出的 buffer 会让处于 pipe 另一端的程序 (例如 head) 无法读到数据, 从而导致更多的网络请求发生.

@iambus
Copy link
Owner

iambus commented Aug 16, 2014

可以用lx list 0-9

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants