flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "hui.pan" <hui....@wbkit.com>
Subject Re: A puzzy problem about flume Failover Sink Processor
Date Mon, 03 Apr 2017 10:25:56 GMT
<html>
<head>
    <meta http-equiv='Content-Type' content='text/html; charset=UTF-8'>
</head>
<body>
<style>
    font{
        line-height: 1.5;
    }
</style>
<div style = 'font-family:"微软雅黑"; font-size: 14px; color:#000000; line-height:1.5;'>
    <div>
<div>Hi:</div><div>&nbsp; &nbsp; &nbsp;Thank you for your advise,and
I will have a try.</div><div><br></div><div>Regrads,</div>
<div><span style="line-height: 1.5;">Hollis Pan</span></div><div
id="ntes-pcmail-signature" style="font-family:'微软雅黑'"><font style="padding:
0; margin:0;">                </font>

</div><br>
</div><div class="J-reply" style="background-color:#f2f2f2;color:black;padding-top:6px;padding-bottom:6px;border-radius:3px;-moz-border-radius:3px;-webkit-border-radius:3px;margin-top:45px;margin-bottom:20px;font-family:'微软雅黑';">
    <div style="font-size:14px;line-height:1.5;word-break:break-all;margin-left:10px;margin-right:10px">On
<span class="mail-date">04/1/2017 21:18</span>,<a class="mail-to" style="text-decoration:none;color:#2a97ff;"
href="mailto:denes@cloudera.com">Denes Arvay&lt;denes@cloudera.com&gt;</a>
wrote: </div>
</div>
<blockquote id="ntes-pcmail-quote" style="margin: 0; padding: 0; font-size: 13px; font-family:
'微软雅黑';">
<div dir="ltr"><div dir="ltr" class="gmail_msg">Hi,<div class="gmail_msg"><br
class="gmail_msg"></div><div class="gmail_msg">If you are using HDFS HA you
don't have to do any special configuration on Flume side. One HDFS Sink is enough, just use
the logical nameservice id in the hdfs.path property instead of the host name.</div><div
class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">See:&nbsp;<a
href="https://community.cloudera.com/t5/Data-Ingestion-Integration/Flume-HDFS-HA/td-p/29141"
class="gmail_msg" target="_blank">https://community.cloudera.com/t5/Data-Ingestion-Integration/Flume-HDFS-HA/td-p/29141</a><br
class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div><div
class="gmail_msg">Regards,</div><div class="gmail_msg">Denes</div></div><div
dir="ltr" class="gmail_msg"><div class="gmail_msg"><br class="gmail_msg"><div
class="gmail_quote gmail_msg"><div dir="ltr" class="gmail_msg">On Sat, Apr 1, 2017
at 12:31 PM <a href="mailto:hui.pan@wbkit.com" class="gmail_msg" target="_blank">hui.pan@wbkit.com</a>
&lt;<a href="mailto:hui.pan@wbkit.com" class="gmail_msg" target="_blank">hui.pan@wbkit.com</a>&gt;
wrote:<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_msg">
<div class="gmail_msg"><span class="gmail_msg"></span><span style="color:rgb(0,0,0);background-color:rgba(0,0,0,0)"
class="gmail_msg">&nbsp; I&nbsp;have&nbsp;a&nbsp;puzzy&nbsp;problem&nbsp;about&nbsp;flume&nbsp;Failover&nbsp;Sink&nbsp;Processor,Description&nbsp;as&nbsp;below:<br
class="gmail_msg">&nbsp;&nbsp;Enviroment&nbsp;is&nbsp;that&nbsp;we&nbsp;do&nbsp;HA&nbsp;configuration&nbsp;in&nbsp;HDFS,that&nbsp;is&nbsp;we&nbsp;have&nbsp;2&nbsp;namenode,1&nbsp;status&nbsp;for&nbsp;active(host&nbsp;name:n1)&nbsp;and&nbsp;another&nbsp;for&nbsp;standby(hostname:n2).<br
class="gmail_msg">&nbsp;&nbsp;We&nbsp;configure&nbsp;our&nbsp;flume.conf&nbsp;like&nbsp;this:<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;#&nbsp;Flume&nbsp;agent&nbsp;config<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sources&nbsp;=&nbsp;source-cd&nbsp;<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.channels&nbsp;=&nbsp;c_cd&nbsp;<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinks&nbsp;=&nbsp;s_cd1&nbsp;&nbsp;s_cd2&nbsp;&nbsp;//s_cd1's&nbsp;hostname&nbsp;is&nbsp;n1,s_cd2's&nbsp;hotname&nbsp;is&nbsp;n2<br
class="gmail_msg"><br class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinkgroups&nbsp;=&nbsp;g1&nbsp;<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinkgroups.g1.sinks&nbsp;=&nbsp;s_cd1&nbsp;s_cd2&nbsp;<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinkgroups.g1.processor.type&nbsp;=&nbsp;failover<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinkgroups.g1.processor.priority.s_cd1&nbsp;=&nbsp;5<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinkgroups.g1.processor.priority.s_cd2&nbsp;=&nbsp;10<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinkgroups.g1.processor.maxpenalty&nbsp;=&nbsp;10000<br
class="gmail_msg"><br class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sources.source-cd.type&nbsp;=&nbsp;org.apache.flume.source.taildir.TaildirSource<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sources.source-cd.channels&nbsp;=&nbsp;c_cd<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;<br class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.channels.c_cd.type&nbsp;=&nbsp;memory<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.channels.c_cd.capacity&nbsp;=&nbsp;5000000<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.channels.c_cd.transactionCapacity&nbsp;=&nbsp;1000<br
class="gmail_msg"><br class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinks.s_cd1.type&nbsp;=&nbsp;hdfs<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinks.s_cd1.hdfs.path&nbsp;=&nbsp;hdfs://<a
href="http://192.168.1.31:8020/PATH" class="gmail_msg" target="_blank">192.168.1.31:8020/PATH</a><br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinks.s_cd1.channel&nbsp;=&nbsp;c_cd<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinks.s_cd1.hdfs.batchSize&nbsp;=&nbsp;1000<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinks.s_cd1.hdfs.useLocalTimeStamp&nbsp;=&nbsp;true<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinks.s_cd1.hdfs.rollSize&nbsp;=&nbsp;0<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinks.s_cd1.hdfs.rollCount&nbsp;=&nbsp;1000000<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinks.s_cd1.hdfs.rollInterval&nbsp;=&nbsp;600<br
class="gmail_msg"><br class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinks.s_cd2.type&nbsp;=&nbsp;hdfs<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinks.s_cd2.hdfs.path&nbsp;=&nbsp;hdfs://<a
href="http://192.168.1.32:8020/PATH" class="gmail_msg" target="_blank">192.168.1.32:8020/PATH</a><br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinks.s_cd2.channel&nbsp;=&nbsp;c_cd<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinks.s_cd2.hdfs.batchSize&nbsp;=&nbsp;1000<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinks.s_cd2.hdfs.useLocalTimeStamp&nbsp;=&nbsp;true&nbsp;<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinks.s_cd2.hdfs.rollSize&nbsp;=&nbsp;0<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinks.s_cd2.hdfs.rollCount&nbsp;=&nbsp;1000000<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;c1.sinks.s_cd2.hdfs.rollInterval&nbsp;=&nbsp;600<br
class="gmail_msg">&nbsp;<br class="gmail_msg">&nbsp;&nbsp;&nbsp;In&nbsp;the&nbsp;flume.conf,we&nbsp;know&nbsp;that&nbsp;the&nbsp;priority&nbsp;of&nbsp;n2&nbsp;is&nbsp;higher&nbsp;than&nbsp;n1,so&nbsp;data&nbsp;should&nbsp;be&nbsp;put&nbsp;into&nbsp;n2.Unfortunately,n2&nbsp;is&nbsp;breakdown&nbsp;when&nbsp;run&nbsp;big&nbsp;data.according&nbsp;to&nbsp;flume&nbsp;<br
class="gmail_msg">&nbsp;mechanism,unfinished&nbsp;data&nbsp;should&nbsp;be&nbsp;flow&nbsp;to&nbsp;n1,but&nbsp;the&nbsp;result&nbsp;is&nbsp;not&nbsp;like&nbsp;this.The&nbsp;log&nbsp;is&nbsp;like&nbsp;this:<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br class="gmail_msg"><br
class="gmail_msg">2017-04-01&nbsp;15:31:28,285&nbsp;WARN&nbsp;org.apache.hadoop.hdfs.LeaseRenewer:&nbsp;Failed&nbsp;to&nbsp;renew&nbsp;lease&nbsp;for&nbsp;[DFSClient_NONMAPREDUCE_-1862131888_34]&nbsp;for&nbsp;525&nbsp;seconds.&nbsp;&nbsp;Will&nbsp;retry&nbsp;shortly&nbsp;...<br
class="gmail_msg">java.net.ConnectException:&nbsp;Call&nbsp;From&nbsp;<a
href="http://archive.cloudera.com/192.168.1.31" class="gmail_msg" target="_blank">archive.cloudera.com/192.168.1.31</a>&nbsp;to&nbsp;slave01:8020&nbsp;failed&nbsp;on&nbsp;connection&nbsp;exception:&nbsp;java.net.ConnectException:&nbsp;Connection&nbsp;refused;&nbsp;For&nbsp;more&nbsp;details&nbsp;see:&nbsp;&nbsp;<a
href="http://wiki.apache.org/hadoop/ConnectionRefused" class="gmail_msg" target="_blank">http://wiki.apache.org/hadoop/ConnectionRefused</a><br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;sun.reflect.GeneratedConstructorAccessor6.newInstance(Unknown&nbsp;Source)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;java.lang.reflect.Constructor.newInstance(Constructor.java:526)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.ipc.Client.call(Client.java:1475)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.ipc.Client.call(Client.java:1408)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;com.sun.proxy.$Proxy20.renewLease(Unknown&nbsp;Source)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:576)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;sun.reflect.GeneratedMethodAccessor6.invoke(Unknown&nbsp;Source)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;java.lang.reflect.Method.invoke(Method.java:606)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;com.sun.proxy.$Proxy21.renewLease(Unknown&nbsp;Source)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:922)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;java.lang.Thread.run(Thread.java:745)<br
class="gmail_msg">Caused&nbsp;by:&nbsp;java.net.ConnectException:&nbsp;Connection&nbsp;refused<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;sun.nio.ch.SocketChannelImpl.checkConnect(Native&nbsp;Method)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:713)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.ipc.Client.getConnection(Client.java:1524)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.ipc.Client.call(Client.java:1447)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>...&nbsp;16&nbsp;more<br
class="gmail_msg"><br class="gmail_msg">&nbsp;According&nbsp;to&nbsp;this&nbsp;log,n2&nbsp;is&nbsp;breakdown&nbsp;meanwhile&nbsp;n1&nbsp;doesn't&nbsp;get&nbsp;any&nbsp;data.So&nbsp;we&nbsp;recovery&nbsp;n2.and&nbsp;the&nbsp;log&nbsp;is&nbsp;like&nbsp;this:<br
class="gmail_msg">&nbsp;&nbsp;&nbsp;<br class="gmail_msg"><br class="gmail_msg">&nbsp;2017-03-31&nbsp;21:16:35,783&nbsp;WARN&nbsp;org.apache.flume.sink.hdfs.HDFSEventSink:&nbsp;HDFS&nbsp;IO&nbsp;error<br
class="gmail_msg">org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):&nbsp;Operation&nbsp;category&nbsp;WRITE&nbsp;is&nbsp;not&nbsp;supported&nbsp;in&nbsp;state&nbsp;standby.&nbsp;Visit&nbsp;<a
href="https://s.apache.org/sbnn-error" class="gmail_msg" target="_blank">https://s.apache.org/sbnn-error</a><br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:88)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1826)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1404)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:592)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.create(AuthorizationProviderProxyClientProtocol.java:111)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:393)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;java.security.AccessController.doPrivileged(Native&nbsp;Method)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;javax.security.auth.Subject.doAs(Subject.java:415)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)<br
class="gmail_msg"><span style="white-space:pre-wrap" class="gmail_msg">	</span>at&nbsp;org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)<br
class="gmail_msg">&nbsp;<br class="gmail_msg">&nbsp;So&nbsp;how&nbsp;can&nbsp;we&nbsp;do&nbsp;to&nbsp;make&nbsp;n1&nbsp;get&nbsp;data&nbsp;when&nbsp;n2&nbsp;is&nbsp;breakdown.Is&nbsp;there&nbsp;any&nbsp;wrong&nbsp;about&nbsp;my&nbsp;configuration&nbsp;or&nbsp;this&nbsp;situation&nbsp;is&nbsp;unsuitable&nbsp;for&nbsp;this.Thanks&nbsp;a&nbsp;lot.<br
class="gmail_msg">&nbsp; &nbsp;</span></div>
<div class="gmail_msg"><br class="gmail_msg"></div><hr style="width:210px;height:1px"
color="#b5c4df" size="1" align="left" class="gmail_msg">
<div class="gmail_msg"><span class="gmail_msg"><div style="MARGIN:10px;FONT-FAMILY:verdana;FONT-SIZE:10pt"
class="gmail_msg"><div class="gmail_msg">Hollis @ Wbkit</div><div class="gmail_msg">Apr
1 .2017</div></div></span></div>
</div></blockquote></div></div></div></div>

</blockquote><style type="text/css">
        a#ntes-pcmail-signature-default:hover {
            text-decoration: underline;
            color: #3593db;
            cursor: pointer;
        }
    </style><!--😀-->
</div>
</body>
</html>

Mime
View raw message