phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject Re: Problem starting region server with Hbase version hbase-2.0.0
Date Wed, 06 Jun 2018 23:11:01 GMT
Thanks all.

in my older version of Hbase 1.2.3 I had added the correct phoenix jar file
(phoenix-4.8.1-HBase-1.2-client.jar) to /lib directory of Hbase.

I found the correct jar file for Hbase 2.0.0
in phoenix-5.0.0-alpha-HBase-2.0-client.jar

jar tvf phoenix-5.0.0-alpha-HBase-2.0-client.jar|grep IndexedWALEditCodec
  1881 Thu Feb 08 17:36:50 GMT 2018
org/apache/hadoop/hbase/regionserver/wal/IndexedWALEditCodec$BinaryCompatibleCompressedIndexKeyValueDecoder.class
  1223 Thu Feb 08 17:36:50 GMT 2018
org/apache/hadoop/hbase/regionserver/wal/IndexedWALEditCodec$BinaryCompatibleIndexKeyValueDecoder.class
   830 Thu Feb 08 17:36:50 GMT 2018
org/apache/hadoop/hbase/regionserver/wal/IndexedWALEditCodec$BinaryCompatiblePhoenixBaseDecoder.class
  1801 Thu Feb 08 17:36:50 GMT 2018
org/apache/hadoop/hbase/regionserver/wal/IndexedWALEditCodec$CompressedIndexKeyValueDecoder.class
  1919 Thu Feb 08 17:36:50 GMT 2018
org/apache/hadoop/hbase/regionserver/wal/IndexedWALEditCodec$CompressedIndexKeyValueEncoder.class
  1143 Thu Feb 08 17:36:50 GMT 2018
org/apache/hadoop/hbase/regionserver/wal/IndexedWALEditCodec$IndexKeyValueDecoder.class
  1345 Thu Feb 08 17:36:50 GMT 2018
org/apache/hadoop/hbase/regionserver/wal/IndexedWALEditCodec$IndexKeyValueEncoder.class
   755 Thu Feb 08 17:36:50 GMT 2018
org/apache/hadoop/hbase/regionserver/wal/IndexedWALEditCodec$PhoenixBaseDecoder.class
   762 Thu Feb 08 17:36:50 GMT 2018
org/apache/hadoop/hbase/regionserver/wal/IndexedWALEditCodec$PhoenixBaseEncoder.class
  4436 Thu Feb 08 17:36:50 GMT 2018
org/apache/hadoop/hbase/regionserver/wal/IndexedWALEditCodec.class

so the region server started OK but then I had a problem with master :(

java.lang.IllegalStateException: The procedure WAL relies on the ability to
hsync for proper operation during component failures, but the underlying
filesystem does not support doing so. Please check the config value of
'hbase.procedure.store.wal.use.hsync' to set the desired level of
robustness and ensure the config value of 'hbase.wal.dir' points to a
FileSystem mount that can provide it.


I tried that mentioned property in hbase-site.xml but no luck. However, I
saw this recent note
<https://stackoverflow.com/questions/50229580/hbase-shell-cannot-use-error-keepererrorcode-nonode-for-hbase-master>


   - I had similar issues with the recent HBase 2.x beta releases, whereas
   everything was OK with stable 1.x releases. Are you using 2.x beta? –
   VS_FF <https://stackoverflow.com/users/7241513/vs-ff> May 8 at 13:16
   <https://stackoverflow.com/questions/50229580/hbase-shell-cannot-use-error-keepererrorcode-nonode-for-hbase-master#comment87486468_50229580>
   -
   yes, i guess that it is caused by releases problem – Solodye
   <https://stackoverflow.com/users/8351601/solodye> May 10 at 12:58
   <https://stackoverflow.com/questions/50229580/hbase-shell-cannot-use-error-keepererrorcode-nonode-for-hbase-master#comment87563798_50229580>


So I revered back to stable release Hbase 1.2.6 unless someone has resolved
this issue.

Thanks

Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 6 June 2018 at 23:24, Juan Jose Escobar <juanjose.escobar@gmail.com>
wrote:

> Hello Mich,
>
> Verify you have the right jars (from your commnts I guess should be
> phoenix-5.0.0-alpha-HBase-2.0-server.jar), that it shows in HBase
> classpath
> and that it contains the missing class e.g. with jar -vtf.
>
> Also, check if there are any pending WALs that are making the startup fail,
> I had similar problem and Phoenix seemed to cause problems at startup until
> I removed the WALs.
>
>
>
>
>
>
>
>
> On Wed, Jun 6, 2018 at 10:55 PM, Mich Talebzadeh <
> mich.talebzadeh@gmail.com>
> wrote:
>
> > Thanks Sean. I downloaded Phoenix for Hbase version 2
> > (apache-phoenix-5.0.0-alpha-HBase-2.0-bin) but still the same error
> >
> > 2018-06-06 21:45:15,297 INFO  [regionserver/rhes75:16020]
> > wal.AbstractFSWAL: WAL configuration: blocksize=256 MB, rollsize=128 MB,
> > prefix=rhes75%2C16020%2C1528317910703, suffix=,
> > logDir=hdfs://rhes75:9000/hbase/WALs/rhes75,16020,152
> > 8317910703, archiveDir=hdfs://rhes75:9000/hbase/oldWALs
> > 2018-06-06 21:45:15,414 ERROR [regionserver/rhes75:16020]
> > regionserver.HRegionServer: ***** ABORTING region server
> > rhes75,16020,1528317910703: Unhandled: Unable to find
> > org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec **
> > ***
> >
> > *java.lang.UnsupportedOperationException: Unable to find
> > org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec*        at
> > org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(
> > ReflectionUtils.java:47)
> >         at
> > org.apache.hadoop.hbase.regionserver.wal.WALCellCodec.
> > create(WALCellCodec.java:112)
> >         at
> > org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.
> > getCodec(AbstractProtobufLogWriter.java:75)
> >         at
> > org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.
> > initAfterHeader0(AbstractProtobufLogWriter.java:184)
> >         at
> > org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.
> > initAfterHeader(AbstractProtobufLogWriter.java:192)
> >         at
> > org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(
> > AbstractProtobufLogWriter.java:174)
> >         at
> > org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(
> > AsyncFSWALProvider.java:99)
> >         at
> > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.
> createWriterInstance(
> > AsyncFSWAL.java:612)
> >         at
> > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.
> createWriterInstance(
> > AsyncFSWAL.java:124)
> >         at
> > org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(
> > AbstractFSWAL.java:759)
> >         at
> > org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(
> > AbstractFSWAL.java:489)
> >         at
> > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.<
> > init>(AsyncFSWAL.java:251)
> >         at
> > org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(
> > AsyncFSWALProvider.java:69)
> >         at
> > org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(
> > AsyncFSWALProvider.java:44)
> >         at
> > org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(
> > AbstractFSWALProvider.java:138)
> >         at
> > org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(
> > AbstractFSWALProvider.java:57)
> >         at
> > org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:252)
> >         at
> > org.apache.hadoop.hbase.regionserver.HRegionServer.
> > getWAL(HRegionServer.java:2065)
> >         at
> > org.apache.hadoop.hbase.regionserver.HRegionServer.
> > buildServerLoad(HRegionServer.java:1291)
> >         at
> > org.apache.hadoop.hbase.regionserver.HRegionServer.
> tryRegionServerReport(
> > HRegionServer.java:1172)
> >         at
> > org.apache.hadoop.hbase.regionserver.HRegionServer.
> > run(HRegionServer.java:989)
> >         at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec
> >         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> >         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> >         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> >         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> >         at java.lang.Class.forName0(Native Method)
> >         at java.lang.Class.forName(Class.java:264)
> >         at
> > org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(
> > ReflectionUtils.java:43)
> >         ... 21 more
> > 2018-06-06 21:45:15,415 ERROR [regionserver/rhes75:16020]
> > regionserver.HRegionServer: RegionServer abort: loaded coprocessors are:
> []
> > 2018-06-06 21:45:15,420 INFO  [regionserver/rhes75:16020]
> > regionserver.HRegionServer:    "Verbose" : false,
> >     "ObjectPendingFinalizationCount" : 0,
> >     "NonHeapMemoryUsage" : {
> >       "committed" : 59793408,
> >       "init" : 2555904,
> >       "max" : -1,
> >       "used" : 58519176
> >     },
> >     "HeapMemoryUsage" : {
> >       "committed" : 1017708544,
> >       "init" : 1052770304,
> >       "max" : 16777347072,
> >       "used" : 255809352
> >     },
> >     "ObjectName" : "java.lang:type=Memory"
> >   } ],
> >   "beans" : [ {
> >     "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC",
> >     "modelerType" : "RegionServer,sub=IPC",
> >     "tag.Context" : "regionserver",
> >     "tag.Hostname" : "rhes75"
> >   } ],
> >   "beans" : [ {
> >     "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication",
> >     "modelerType" : "RegionServer,sub=Replication",
> >     "tag.Context" : "regionserver",
> >     "tag.Hostname" : "rhes75"
> >   } ],
> >   "beans" : [ {
> >     "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server",
> >     "modelerType" : "RegionServer,sub=Server",
> >     "tag.Context" : "regionserver",
> >     "tag.Hostname" : "rhes75"
> >   } ]
> > }
> > 2018-06-06 21:45:15,430 INFO  [regionserver/rhes75:16020]
> > regionserver.HRegionServer: ***** STOPPING region server
> > 'rhes75,16020,1528317910703' *****
> > 2018-06-06 21:45:15,430 INFO  [regionserver/rhes75:16020]
> > regionserver.HRegionServer: STOPPED: Unhandled: Unable to find
> > org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec
> > 2018-06-06 21:45:15,430 INFO  [regionserver/rhes75:16020]
> > regionserver.SplitLogWorker: Sending interrupt to stop the worker thread
> > 2018-06-06 21:45:15,430 INFO  [regionserver/rhes75:16020]
> > regionserver.HRegionServer: Stopping infoServer
> > 2018-06-06 21:45:15,430 INFO  [SplitLogWorker-rhes75:16020]
> > regionserver.SplitLogWorker: SplitLogWorker interrupted. Exiting.
> > 2018-06-06 21:45:15,430 INFO  [SplitLogWorker-rhes75:16020]
> > regionserver.SplitLogWorker: SplitLogWorker rhes75,16020,1528317910703
> > exiting
> > 2018-06-06 21:45:15,434 INFO  [regionserver/rhes75:16020]
> > handler.ContextHandler: Stopped o.e.j.w.WebAppContext@1e530163
> > {/,null,UNAVAILABLE}{file:/data6/hduser/hbase-2.0.0/
> > hbase-webapps/regionserver}
> > 2018-06-06 21:45:15,436 INFO  [regionserver/rhes75:16020]
> > server.AbstractConnector: Stopped ServerConnector@5c60b0a0
> > {HTTP/1.1,[http/1.1]}{0.0.0.0:16030}
> > 2018-06-06 21:45:15,436 INFO  [regionserver/rhes75:16020]
> > handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@39651a82
> > {/static,file:///data6/hduser/hbase-2.0.0/hbase-webapps/
> > static/,UNAVAILABLE}
> > 2018-06-06 21:45:15,436 INFO  [regionserver/rhes75:16020]
> > handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@70211e49
> > {/logs,file:///data6/hduser/hbase-2.0.0/logs/,UNAVAILABLE}
> > 2018-06-06 21:45:15,437 INFO  [regionserver/rhes75:16020]
> > regionserver.HeapMemoryManager: Stopping
> > 2018-06-06 21:45:15,437 INFO  [regionserver/rhes75:16020]
> > flush.RegionServerFlushTableProcedureManager: Stopping region server
> flush
> > procedure manager abruptly.
> > 2018-06-06 21:45:15,437 INFO  [MemStoreFlusher.1]
> > regionserver.MemStoreFlusher: MemStoreFlusher.1 exiting
> > 2018-06-06 21:45:15,437 INFO  [regionserver/rhes75:16020]
> > snapshot.RegionServerSnapshotManager: Stopping
> RegionServerSnapshotManager
> > abruptly.
> > 2018-06-06 21:45:15,437 INFO  [MemStoreFlusher.0]
> > regionserver.MemStoreFlusher: MemStoreFlusher.0 exiting
> > 2018-06-06 21:45:15,437 INFO  [regionserver/rhes75:16020]
> > regionserver.HRegionServer: aborting server rhes75,16020,1528317910703
> > 2018-06-06 21:45:15,437 INFO  [regionserver/rhes75:16020]
> > zookeeper.ReadOnlyZKClient: Close zookeeper connection 0x2a9ccc02 to
> > localhost:2181
> > 2018-06-06 21:45:15,439 INFO  [regionserver/rhes75:16020]
> > regionserver.HRegionServer: stopping server rhes75,16020,1528317910703;
> all
> > regions closed.
> > 2018-06-06 21:45:15,440 INFO  [regionserver/rhes75:16020]
> > regionserver.Leases: Closed leases
> > 2018-06-06 21:45:15,440 INFO  [regionserver/rhes75:16020]
> > hbase.ChoreService: Chore service for: regionserver/rhes75:16020 had
> > [[ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit:
> > MILLISECONDS], [ScheduledChore: Nam
> > e: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS],
> > [ScheduledChore: Name: MovedRegionsCleaner for region
> > rhes75,16020,1528317910703 Period: 120000 Unit: MILLISECONDS],
> > [ScheduledChore: Name: MemstoreFlusherChore Period: 1
> > 0000 Unit: MILLISECONDS]] on shutdown
> > 2018-06-06 21:45:15,440 INFO  [regionserver/rhes75:16020.logRoller]
> > regionserver.LogRoller: LogRoller exiting.
> >
> >
> > Dr Mich Talebzadeh
> >
> >
> >
> > LinkedIn * https://www.linkedin.com/profile/view?id=
> > AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCd
> > OABUrV8Pw>*
> >
> >
> >
> > http://talebzadehmich.wordpress.com
> >
> >
> > *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> > loss, damage or destruction of data or any other property which may arise
> > from relying on this email's technical content is explicitly disclaimed.
> > The author will in no case be liable for any monetary damages arising
> from
> > such loss, damage or destruction.
> >
> >
> >
> > On 6 June 2018 at 20:49, Sean Busbey <busbey@apache.org> wrote:
> >
> > > IndexedWALEditCodec is a class from the Apache Phoenix project. your
> > > cluster must be configured to have Phoenix run but it can't find the
> > > jars for phoenix.
> > >
> > > user@phoenix.apache.org is probably your best bet for getting things
> > > going.
> > >
> > > On Wed, Jun 6, 2018 at 1:52 PM, Mich Talebzadeh
> > > <mich.talebzadeh@gmail.com> wrote:
> > > > Hi,
> > > >
> > > > I have an old Hbase hbase-1.2.3 that runs fine on both RHES 5.6 and
> > RHES
> > > 7.5
> > > >
> > > > I created a new Hbase hbase-2.0.0 instance on RHES 7.5.
> > > >
> > > > I seem to have a problem with my region server as it fails to start
> > > > throwing error
> > > >
> > > > 2018-06-06 19:28:37,033 INFO  [regionserver/rhes75:16020]
> > > > regionserver.HRegionServer: CompactionChecker runs every PT10S
> > > > 2018-06-06 19:28:37,071 INFO  [SplitLogWorker-rhes75:16020]
> > > > regionserver.SplitLogWorker: SplitLogWorker
> rhes75,16020,1528309715572
> > > > starting
> > > > 2018-06-06 19:28:37,073 INFO  [regionserver/rhes75:16020]
> > > > regionserver.HeapMemoryManager: Starting, tuneOn=false
> > > > 2018-06-06 19:28:37,076 INFO  [regionserver/rhes75:16020]
> > > > regionserver.ChunkCreator: Allocating data MemStoreChunkPool with
> chunk
> > > > size 2 MB, max count 2880, initial count 0
> > > > 2018-06-06 19:28:37,077 INFO  [regionserver/rhes75:16020]
> > > > regionserver.ChunkCreator: Allocating index MemStoreChunkPool with
> > chunk
> > > > size 204.80 KB, max count 3200, initial count 0
> > > > 2018-06-06 19:28:37,078 INFO  [ReplicationExecutor-0]
> > > > regionserver.ReplicationSourceManager: Current list of replicators:
> > > > [rhes75,16020,1528309715572] other RSs: [rhes75,16020,1528309715572]
> > > > 2018-06-06 19:28:37,099 INFO  [regionserver/rhes75:16020]
> > > > regionserver.HRegionServer: Serving as rhes75,16020,1528309715572,
> > > > RpcServer on rhes75/50.140.197.220:16020,
> sessionid=0x163d61b308c0033
> > > > 2018-06-06 19:28:37,100 INFO  [regionserver/rhes75:16020]
> > > > quotas.RegionServerRpcQuotaManager: Quota support disabled
> > > > 2018-06-06 19:28:37,100 INFO  [regionserver/rhes75:16020]
> > > > quotas.RegionServerSpaceQuotaManager: Quota support disabled, not
> > > starting
> > > > space quota manager.
> > > > 2018-06-06 19:28:40,133 INFO  [regionserver/rhes75:16020]
> > > > wal.AbstractFSWAL: WAL configuration: blocksize=256 MB, rollsize=128
> > MB,
> > > > prefix=rhes75%2C16020%2C1528309715572, suffix=,
> > > > logDir=hdfs://rhes75:9000/hbase/WALs/rhes75,16020,152
> > > > 8309715572, archiveDir=hdfs://rhes75:9000/hbase/oldWALs
> > > > 2018-06-06 19:28:40,251 ERROR [regionserver/rhes75:16020]
> > > > regionserver.HRegionServer: ***** ABORTING region server
> > > > rhes75,16020,1528309715572: Unhandled: Unable to find
> > > > org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec **
> > > > ***
> > > >
> > > > I cannot seem to be able to fix this even after removing hbase
> > directory
> > > > from hdfs and zookeeper! Any ideas will be appreciated.
> > > >
> > > > Thanks
> > > >
> > > > Dr Mich Talebzadeh
> > > >
> > > >
> > > >
> > > > LinkedIn * https://www.linkedin.com/profile/view?id=
> > > AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> > > > <https://www.linkedin.com/profile/view?id=
> > AAEAAAAWh2gBxianrbJd6zP6AcPCCd
> > > OABUrV8Pw>*
> > > >
> > > >
> > > >
> > > > http://talebzadehmich.wordpress.com
> > > >
> > > >
> > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for
> > any
> > > > loss, damage or destruction of data or any other property which may
> > arise
> > > > from relying on this email's technical content is explicitly
> > disclaimed.
> > > > The author will in no case be liable for any monetary damages arising
> > > from
> > > > such loss, damage or destruction.
> > >
> >
>

Mime
View raw message