Hi,

@Mike thanks for ur reply.

1) After executing Flume-ng agent, and avro client, File is created in HDFS.
I used today same flume-ng setup with hadoop 1.0.1.
Now i m facing problem that through avro client i am sending normal text file. But inside HDFS File content is coming like as shown below. I want in HDFS this file content should be in normal text format
HDFS File Content:
"SEQ^F!org.apache.hadoop.io.LongWritable"org.apache.hadoop.io.BytesWritable^@^@^@^@^@^@^UG^O~vz/<87>^[~^@^@^@)^@^@^@^H^@^@^A8[<8e>)^@^@^@^]We are modifying the file now^@^@^@

Given txt file content through AvroClient is
We are modifying the file now

Kindly provide ur inputs to resolve this issue.
my flume.conf file content is as folows:
# Define a memory channel called ch1 on agent1
agent1.channels.ch1.type = memory


# Define an Avro source called avro-source1 on agent1 and tell it
# to bind to 0.0.0.0:41414. Connect it to channel ch1.
agent1.sources.avro-source1.channels = ch1
agent1.sources.avro-source1.type = avro
agent1.sources.avro-source1.selector.type=replicating
agent1.sources.avro-source1.bind = 0.0.0.0
agent1.sources.avro-source1.port = 41414


# Define a hdfs sink that simply logs all events it receives
# and connect it to the other end of the same channel.
agent1.sinks.HDFS.channel = ch1
agent1.sinks.HDFS.type = hdfs
agent1.sinks.HDFS.hdfs.path = hdfs://localhost:54310/user/hadoop-node1/flumeTest
agent1.sinks.HDFS.hdfs.file.Type = DataStream
agent1.sinks.HDFS.hdfs.file.Format = Text

# Finally, now that we've defined all of our components, tell
# agent1 which ones we want to activate.
agent1.channels = ch1
agent1.sources = avro-source1
agent1.sinks = HDFS



2) AT Flume NG Side still i am getting security related IO Exception. when i start flume-ng using above configuration file.
Exception log coming at flume-ng side is :
2012-07-06 11:14:42,957 (conf-file-poller-0) [DEBUG - org.apache.hadoop.security.Groups.<init>(Groups.java:59)] Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
2012-07-06 11:14:42,961 (conf-file-poller-0) [DEBUG - org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227)] java.io.IOException: config()
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227)
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:214)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:187)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:239)
at org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:212)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:187)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:239)
at org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:516)
at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:239)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSinks(PropertiesFileConfigurationProvider.java:373)
at org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:223)
at org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
at org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
at org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)




With Regards,
Amit Handa


On Fri, Jul 6, 2012 at 12:21 AM, Mike Percy <mpercy@cloudera.com> wrote:
On Thu, Jul 5, 2012 at 12:28 AM, Amit Handa <amithanda01@gmail.com> wrote:
HI All,

While trying to run Flume ng using HDFS SInk, and using avro Client.. i am getting IOException. Kindly help in resolving this issue

Exception log is as follows:
2012-07-05 12:01:32,789 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)] Creating instance of sink HDFS typehdfs
2012-07-05 12:01:32,816 (conf-file-poller-0) [DEBUG - org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227)] java.io.IOException: config()
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227)
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:214)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:187)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:239)
...

Nothing is wrong with this, you are running at DEBUG level and Hadoop is giving you debug-level output. If you don't want to get DEBUG level messages from Hadoop while running Flume at DEBUG level then you will need to add something like:

log4j.logger.org.apache.hadoop = INFO

To your log4j.properties file.

Are you experiencing any problems with your setup?

Regards,
Mike