flume包安装手册

flume包安装手册 Flume安装手册 1. 环境 linux系统 2. 安装步骤 2.1. 检查安装环境 首先请检查系统是Linux系统安装 首先将需要安装的文件包apache-flume-1.4.0-bin，或者是该文件的压缩包用解压缩命令解压到指定安装目录。 然后到apache-flume-1.4.0-bin/conf目录下修改配置文件flume-conf.properties，如下说明(主要修改consumer部分): ############################################ # producer config ########################################### #agent section producer.sources = s producer.channels = c producer.sinks = r #source section producer.sources.s.type = avro producer.sources.s.channels = c producer.sources.s.bind = 10.243.0.250 producer.sources.s.port = 44443 # Each sink's type must be defined producer.sinks.r.type = org.apache.flume.plugins.KafkaSink producer.sinks.r.metadata.broker.list=10.243.0.250:11110 producer.sinks.r.partition.key=0 producer.sinks.r.partitioner.class=org.apache.flume.plugins.SinglePartition producer.sinks.r.serializer.class=kafka.serializer.StringEncoder producer.sinks.r.request.required.acks=0 producer.sinks.r.max.message.size=1000000 producer.sinks.r.producer.type=sync producer.sinks.r.custom.encoding=UTF-8 producer.sinks.r.custom.topic.name=3test producer.sinks.r.hostname = 10.243.0.250 #Specify the channel the sink should use producer.sinks.r.channel = c # Each channel's type is defined. producer.channels.c.type = memory producer.channels.c.capacity = 1000 ############################################ # consumer config ########################################### consumer.sources = s consumer.channels = c consumer.sinks = r consumer.sources.s.type = avro //对客户端为java代码时设置为:avro,若为C设置为: syslogudp. consumer.sources.s.bind = 10.243.0.250 //flume的主机IP consumer.sources.s.port = 44443 //提供给用户的端口号 consumer.sources.s.channels = c consumer.sinks.r.channel = c consumer.channels.c.type = memory consumer.channels.c.capacity = 1000000 consumer.sinks.r.type = org.apache.flume.plugins.KafkaSink consumer.sinks.r.metadata.broker.list=10.243.0.250:11110,10.243.0.251:11110,10.243.0 .252:11110 //Kafka的IP及端口。 consumer.sinks.r.partition.key=0 consumer.sinks.r.partitioner.class=org.apache.flume.plugins.SinglePartition consumer.sinks.r.serializer.class=kafka.serializer.StringEncoder consumer.sinks.r.request.required.acks=0 consumer.sinks.r.max.message.size=1000000 consumer.sinks.r.producer.type=sync consumer.sinks.r.custom.encoding=UTF-8 consumer.sinks.r.custom.topic.name=p2plog //kafka预定的topic名称 consumer.sinks.r.hostname =10.243.0.250 //kafka的主机 consumer.sources.s.interceptors = i1 consumer.sources.s.interceptors.i1.type = timestamp 说明:虽然kafka和flume安装在同一台服务器上，其主机IP最好不要写成localhost， 而应该写成其对应IP地址。这样便于访问。 2.2. 启动服务 安装完软件包并修改完配置文件后，进入apache-flume-1.4.0-bin/apache-flume-1.4.0-bin目录下，用如下命令启动flume服务: Nohup bin/flume-ng agent --conf conf --conf-file conf/flume-conf.properties --name consumer -Dflume.root.logger=INFO,console& 让flume服务在后台运行，可以通过查看日志nohup.out文件或者是查看进程命令查看 服务是否启动成功。