Can anyone setup a startup documents for this project

[wxstorm]

I have read the readme and api documents, but I can not find any document about how to configure and start up server. And the example config seems to be overdued.
Thanks.

[bpodgursky]

Yeah, this is definitely badly needed. It requires a bit of knowledge about the moving parts to get started. Hopefully we can do this sometime soon.

[wxstorm]

To start hank-server, Is it as follows?
in directory hank-server/bin

1. ./hank-server conductor start ../conf/ring_group_conductor.yaml ../conf/ring_group_conductor.log4j.properties
2. ./hank-server server start ../conf/partition_server.yaml ../conf/partition_server.log4j.properties

configuration files are copied from example_config dir with some changes:

ring_group_conductor.yaml:
ring_group_conductor:

ring_group_name: rg1

sleep_interval: 1000
min_ring_fully_serving_observations: 10
initial_mode: INACTIVE
coordinator:
factory: com.liveramp.hank.coordinator.zk.ZooKeeperCoordinator$Factory
options:

connect_string: localhost:2181
session_timeout: 2000
domains_root: /hank/domains
domain_groups_root: /hank/domain_groups
ring_groups_root: /hank/ring_groups

partition_server.yaml:

partition_server:
local_data_dirs:
- /home/ssd3/data/query
service_port: 12345
ring_group_name: rg1
partition_server_daemon:
num_concurrent_queries: 5
num_concurrent_get_bulk_tasks: 256
get_bulk_task_size: 64
get_timer_aggregator_window: 1000
buffer_reuse_max_size: 1024
cache_num_bytes_capacity: 1000000
cache_num_items_capacity: 2000
num_worker_threads: 256
update_daemon:
num_concurrent_updates: 1
max_concurrent_updates_per_data_directory: 2
coordinator:
factory: com.liveramp.hank.coordinator.zk.ZooKeeperCoordinator$Factory
options:

connect_string: localhost:2181
session_timeout: 2000
domains_root: /hank/domains
domain_groups_root: /hank/domain_groups
ring_groups_root: /hank/ring_groups

BUT, when start conductor， exception is throwed:

2017-07-13 13:46:56,159 INFO  ring_group_conductor.RingGroupConductor (RingGroupConductor.java:run(69)) - Ring Group Conductor for ring group rg1 starting.
2017-07-13 13:46:56,163 ERROR ring_group_conductor.RingGroupConductor (RingGroupConductor.java:run(98)) - unexpected exception!
java.lang.NullPointerException
        at com.liveramp.hank.ring_group_conductor.RingGroupConductor.run(RingGroupConductor.java:74)
        at com.liveramp.hank.ring_group_conductor.RingGroupConductor.main(RingGroupConductor.java:156)
2017-07-13 13:46:56,164 INFO  ring_group_conductor.RingGroupConductor (RingGroupConductor.java:run(102)) - Ring Group Conductor for ring group rg1 shutting down.

what are the reasons? Thanks.

[wxstorm]

The problem has resolved. I need to create ring group rg1 in hank-ui first. then start partition server.
BUT I still can not find how to load data into partition server.
Conductor has been started, But I do not know how to issue load command.
Any tips? @bpodgursky , Appreciate for any help.

Any detailed documents about how to use HANK are appreciated.
3ks.

[bpodgursky]

@wxstorm sorry for the slow replies. Glad that you got that running.

To get a partition server to load data, you need (1) a domain (2) a base built for the domain (3) the domain assigned to the domain group and (4) the RGC to be ACTIVE so it can tell the partition servers to update.

The domain configuration is a bit intimidating looking. The configuration could look something like

Num partitions 1024
Partitioner com.liveramp.hank.partitioner.Murmur64Partitioner
Storage engine factory com.liveramp.hank.storage.cueball.Cueball$Factory

---
key_hash_size: 10
hasher: com.liveramp.hank.hasher.Murmur64Hasher
max_allowed_part_size: 527343750
hash_index_bits: 13
remote_domain_root: /data/hank/my_domain
file_ops_factory: com.liveramp.hank.storage.HdfsPartitionRemoteFileOps$Factory
value_size: 16
num_remote_leaf_versions_to_keep: 2
partition_cache_capacity: 100

The domains are all built via MapReduce jobs currently. Hopefully you can get a sense of how they work from the tests for the two ways of running these, via raw mapreduce

https://github.com/LiveRamp/hank/blob/master/hank-server/src/test/java/com/liveramp/hank/hadoop/TestHadoopDomainBuilder.java

or using Cascading.

https://github.com/LiveRamp/hank/blob/master/hank-server/src/test/java/com/liveramp/hank/cascading/TestCascadingDomainBuilder.java

The basic idea is that the domain builder builds the domains to HDFS, and then the partition server downloads the files from there onto the local disk.

This is pretty high level, so let me know what isn't clear and I can try to help. Sorry the documentation is so sparse / nonexistent.

[wxstorm]

@bpodgursky Thank you very much for you kindness. I have review the DomainBuilder src code, and
still have several questions:

1. Is there any tool or commands to issue the domain-base building MR? where to configure hadoop clusters related informations such as ugi, hadoop address etc.?
2. where to execute the domain assignment action? I have created only one domain group , the domain seems to be automatically included in the only domain group. Does the domain assignment action need some special steps? can you kindly explain the meaning of domain, domain group , ring, ring group and the relations among them? Any use cases? 3ks.

3. when the domain's HDFS data is ready, how to notify RGC to load it?
4. in the ring group , no domain can be selected, any problems?
   

Thanks very much.

[bpodgursky]

(1) setting up a hadoop cluster is a bit out of scope here, but there are a lot of available resources for this. once you have a machine working as a hadoop gateway, the domain builder code can be run there with a standard 'hadoop jar command'.
(2) you need to have a version built before you can attach it to the domain group
(3) if the RGC is active, it will happen automatically.
(4) once you've added the domain to the domain group, it will show up in the ring group.

[wxstorm]

@bpodgursky many thanks. I have understood the load process. And In the ring, the domain appears.
But when partition server executes updating, it prints error:
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataOutputStream。

Does something need to configure in conf/hank-env.sh？ When server starts, It prints warnings:
Warning: environment configuration file was not found: ./../conf/hank-env.sh

What does this file look like， any examples?
Thanks.

[bpodgursky]

Could you put the hadoop classes on the classpath and try? Ex this is what we're running in our init scripts

DATE=$(date +%F-%H.%M.%S)
CLASSPATH=$(hadoop classpath):/etc/hadoop/conf:$HANK_SERVER_JOBJAR
OPTS="-Djava.library.path=/opt/cloudera/parcels/CDH/lib/hadoop/lib/native -Xmx8G -Xms8G "\
"-XX:+HeapDumpOnOutOfMemoryError -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps "\
"-XX:+PrintGCDetails -Xloggc:gc_partition_server_$DATE.log -XX:+UseConcMarkSweepGC"
BIN="$JAVA $OPTS -cp $CLASSPATH $HANK_PARTITIONSERVER_MAINCLASS $YAML $LOG4J_PROPERTIES"

I don't think the hank env is necessary. Let me know if this classpath change isn't enough and I'll gather up all the init scripts we have and post those.

[wxstorm]

@bpodgursky Thank you , the classpath problem seems to be solved when I changed scope of hadoop-related dependencies from provided to compile.

Can the project compile in java6? Our hadoop clusters only support java6 runtime to run MR jobs.

[bpodgursky]

Great.

I doubt it fully compiles under Java 6 anymore. You could throw in a compiler plugin to override the parent and try compiling:

      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>2.3.2</version>
        <configuration>
          <source>1.6</source>
          <target>1.6</target>
        </configuration>
      </plugin>

I strongly suspect there will be a few things that don't compile, although probably not anything massive.

[roshan]

@wxstorm can you contact me via email

[davinchia]

@wxstorm feel free to contact me too. @roshan and I will help you out.