Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Table ownership differences can leave folsom inconsistent #30

Open
russelldb opened this issue Jul 20, 2012 · 7 comments
Open

Table ownership differences can leave folsom inconsistent #30

russelldb opened this issue Jul 20, 2012 · 7 comments

Comments

@russelldb
Copy link
Contributor

The history metric creates a new ets table when a new metric is created. The owner of that table is the process that called folsom_metrics:new_history(Name). However, the folsom table is owned by the folsom supervisor. In the case that the process that owns the history exits the history metric table itself crashes, but the entry in the folsom metrics table remains.

Folsom is then in an inconsistent state. Using folsom_metrics_histogram_ets to create (and therefore own) the table would probably help. Ideally folsom should have a single process that owns all ets tables so that there is consistency (a crash takes them all away, they're insulated from calling process crashes.) Better still would be to implement something like the strategy in this article http://steve.vinoski.net/blog/2011/03/23/dont-lose-your-ets-tables/

I'm raising this as a request for comments before I factor such a strategy into folsom. Opinions?

@joewilliams
Copy link
Contributor

Steve's post seems like a good setup and makes sense to me. Yet another item for the folsom to do list. I'll have some time soon if you would like to collaborate on this and the race condition issues.

@russelldb
Copy link
Contributor Author

Yes. That would be good. When we get 1.2 out I've got some time to work on folsom.

@ghost ghost assigned joewilliams Aug 10, 2012
@mmzeeman
Copy link

At zotonic we recently ran into this issue too. Indeed all tables of folsom should be owned by one process.

Automatic bookkeeping of metrics can be implemented by monitoring the process which creates the new metric. When that process dies folsom can do the bookkeeping without any problem.

The crash we get from time to time.

2013-03-20 17:02:52.468 [error] emulator Error in process <0.32714.199> on node 'zotonic001@Lamma' with exit value:
{badarg,[{ets,delete,[26940620904],[]},{folsom_sample_exdec,delete_and_rescale,4,
[{file,"src/folsom_sample_exdec.erl"},{line,122}]},{folsom_sample_exdec,rescale,5,[{file,"src/folsom_sample_exdec.erl"},
{line,106}]},{folsom_sample_exdec..

@joewilliams
Copy link
Contributor

Anyone have an interest in tackling this one?

@sebmaynard
Copy link

Just ran into this one myself! think I'll hold off using histories for now. I'm going to just stick the few things I wanted it for in a queue in a gen_server, but not ideal...

Looking at the table viewer, it seems the Tids are changing, but the histories index isn't getting updated at the same time.

@joewilliams
Copy link
Contributor

Test message, ignore me.

@joewilliams joewilliams removed their assignment Oct 21, 2015
@joewilliams
Copy link
Contributor

Folsom has moved, please resubmit your issue at https://github.com/folsom-project Thanks!

garret-smith pushed a commit to garret-smith/folsom that referenced this issue Nov 18, 2015
A history will survive a process crash without incident since
it is owned by folsom_sup.  This will leave the rest of the folsom
ets tables in a consistent state (all are owned by folsom_sup).
fischerwb added a commit to fischerwb/folsom that referenced this issue May 5, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants