ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alper Tekinalp <al...@evam.com>
Subject Threads got stuck
Date Tue, 25 Oct 2016 07:32:11 GMT
Hi all.

We have 3 servers and cache configuration like:

    <bean class="org.apache.ignite.configuration.CacheConfiguration"
name="DEFAULT">
        <property name="atomicityMode" value="ATOMIC" />
        <property name="cacheMode" value="PARTITIONED" />
        <property name="memoryMode" value="OFFHEAP_TIERED" />
        <property name="backups" value="1" />
        <property name="affinity">
            <bean
class="org.apache.ignite.cache.affinity.fair.FairAffinityFunction">
                <constructor-arg index="0" type="int"
value="#{evamProperties['topology.cache.partition.size']}"/>
            </bean>
        </property>
        <!-- max memory is set to 0 means unlimited -->
        <property name="offHeapMaxMemory" value="0" />
        <property name="writeThrough" value="false" />
        <property name="writeBehindEnabled" value="false" />
        <property name="eagerTtl" value="false"/>
    </bean>

For our worker threads we check heartbeat and if a thread did not sent
heart beat for 10 minutes we consider it as stucked and interrrupt and
recreate it.

As I can see all our worker threads are stucked in cache.put() state and
interrupted and recreated regularly.

What can be the reason we are stucked at put? Following is stacktrace for
interruption error.

javax.cache.CacheException: class
org.apache.ignite.IgniteInterruptedException: Failed to wait for
asynchronous operation permit (thread got interrupted).
        at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1502)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:2021)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1221)
        at
com.intellica.project.helper.ee.ConfigManagerHelperEE.setState(ConfigManagerHelperEE.java:90)
        at
com.intellica.project.helper.ee.StateMachineConfigManagerEEImpl.store(StateMachineConfigManagerEEImpl.java:53)
        at
com.evelopers.unimod.runtime.AbstractEventProcessor.storeConfig(AbstractEventProcessor.java:175)
        at
com.evelopers.unimod.runtime.AbstractEventProcessor.process(AbstractEventProcessor.java:130)
        at
com.evelopers.unimod.runtime.AbstractEventProcessor.process(AbstractEventProcessor.java:80)
        at
com.evelopers.unimod.runtime.ModelEngine.process(ModelEngine.java:199)
        at
com.evelopers.unimod.runtime.StrictHandler.handle(StrictHandler.java:46)
        at
com.intellica.evam.engine.server.worker.AbstractScenarioWorker.runScenarioLogic(AbstractScenarioWorker.java:172)
        at
com.intellica.evam.engine.server.worker.AbstractScenarioWorker.runScenario(AbstractScenarioWorker.java:130)
        at
com.intellica.evam.engine.server.worker.AsyncWorker.processEvent(AsyncWorker.java:156)
        at
com.intellica.evam.engine.server.worker.AsyncWorker.run(AsyncWorker.java:88)
Caused by: class org.apache.ignite.IgniteInterruptedException: Failed to
wait for asynchronous operation permit (thread got interrupted).
        at
org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUtils.java:747)
        at
org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUtils.java:745)
        ... 14 more
Caused by: java.lang.InterruptedException
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
        at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.asyncOpAcquire(GridCacheAdapter.java:4597)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.putAsync(GridCacheAdapter.java:2541)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2215)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1214)
        ... 11 more


-- 
Alper Tekinalp

Software Developer
Evam Streaming Analytics

Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL

Tel:  +90 216 455 01 53 Fax: +90 216 455 01 54
www.evam.com.tr
<http://www.evam.com>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message