flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tinte garcia, Miguel Angel" <miguel.ti...@atos.net>
Subject RE: Writing file to HDFS from Java
Date Mon, 15 Dec 2014 14:31:57 GMT
Ok, I am using Flume embedded agent to insert into HBase instead of HDFS. I am using this configuration:
              properties.put("channel.type", "memory");
              properties.put("channel.capacity", "100000");
              properties.put("channel.transactionCapacity", "1000");
              properties.put("sinks", "hbaseSink");
              properties.put("hbaseSink.hostname", "localhost");
              properties.put("hbaseSink.port", port);
              properties.put("hbaseSink.type", "avro"); // only avro is supported
              properties.put("hbaseSink.channel","memoryChannel");
              properties.put("hbaseSink.table", tableName);
              properties.put("hbaseSink.columnFamily", columnFamily);
              properties.put("hbaseSink.serializer","org.apache.flume.sink.hbase.RegexHbaseEventSerializer");
              properties.put("processor.type", "failover");

Apparently, embedded agent is created correctly:
              try {
                     agent = new EmbeddedAgent("agentHBase");
                     agent.configure(properties);
                     agent.start();
              } catch (final Exception ex) {
                     System.err.println(ex.toString());
                     }
              }

Once it is running, I am trying to connect it through:
client = RpcClientFactory.getDefaultInstance(hostname, port, 10000);

But it is returning a connection error:
“org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, port: 61616 }: RPC
connection error”

Any idea how can I connect to embedded agent?

Thanks

From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
Sent: Tuesday, November 11, 2014 7:44 PM
To: user@flume.apache.org
Cc: user@flume.apache.org
Subject: Re: Writing file to HDFS from Java

Are you talking about the embedded agent? Embedded agent is meant to only write data to another
Flume agent and does not support the HDFS sink. Only Avro Sink is supported.

Thanks,
Hari


On Tue, Nov 11, 2014 at 10:10 AM, Tinte garcia, Miguel Angel <miguel.tinte@atos.net<mailto:miguel.tinte@atos.net>>
wrote:

This e-mail and the documents attached are confidential and intended solely for the addressee;
it may also be privileged. If you receive this e-mail in error, please notify the sender immediately
and destroy it.
As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered
for the message content. Although the sender endeavors to maintain a computer virus-free network,
the sender does not warrant that this transmission is virus-free and will not be liable for
any damages resulting from any virus transmitted.

Este mensaje y los ficheros adjuntos pueden contener información confidencial destinada solamente
a la(s) persona(s) mencionadas anteriormente y pueden estar protegidos por secreto profesional.
Si usted recibe este correo electrónico por error, gracias por informar inmediatamente al
remitente y destruir el mensaje.
Al no estar asegurada la integridad de este mensaje sobre la red, Atos no se hace responsable
por su contenido. Su contenido no constituye ningún compromiso para el grupo Atos, salvo
ratificación escrita por ambas partes.
Aunque se esfuerza al máximo por mantener su red libre de virus, el emisor no puede garantizar
nada al respecto y no será responsable de cualesquiera daños que puedan resultar de una
transmisión de virus.
Mime
View raw message