madlib-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Neki, Atsushi" <neki.atsu...@jp.fujitsu.com>
Subject RE: Multinomial Regression: Failed with a msg "Hessian or gradient is not finite."
Date Thu, 06 Apr 2017 04:50:26 GMT
Frank,

I tried again the following environments:

1) [NG] MADLib 1.10(Compiling From Source) + PG 9.6 (+ Ubuntu 16.04)
    : Failed. Same result with before.
      As an additional information,
      madlib install-check was failed on this environment.
      It looks similar with the below.
       https://issues.apache.org/jira/browse/MADLIB-1068

2) [OK] MADLib 1.9.1(Binary) + PG 9.5 (+ Ubuntu 16.04)
    : Successed.
      Still remain the msg 'WARNING:  Hessian or gradient is not finite.'

3) [NT] MADLib 1.10(Compiling From Source)  + PG 9.4 (+ Ubuntu 16.04)
    : MADLib Build error (so far, I will not report more about that in here)
    > [ 48%] Building CXX object src/ports/postgres/9.6/CMakeFiles/madlib_postgresql_9_6.dir/__/__/__/modules/linalg/matrix_decomp.cpp.o
    > g++: internal compiler error: Killed (program cc1plus)

4) [OK] MADLib 1.10(Compiling From Source)  + PG 9.6 (+ CentOS 7)
    : Successed.
      Still remain the msg 'WARNING:  Hessian or gradient is not finite.'


I will move my workspace to the 4th environment (1.10 + PG9.6 + CentOS).
Thank you very much for your quick response and advice.


Does the msg 'WARNING:  Hessian or gradient is not finite.' mean that something is wrong?
Should I care something about it?
In case I see the message, some NaN values are included in output.
This model can predict category well.


  # select * from mactbl_output;
  -[ RECORD 1 ]------+-----------------------------------------------------------------------
  category           | 602
  coef               | {104.464636547264,1.72398814787594,1.29895329386266,0.688758979636415}
  log_likelihood     | NaN
  std_err            | {NaN,NaN,NaN,NaN}
  z_stats            | {NaN,NaN,NaN,NaN}
  p_values           | {NaN,NaN,NaN,NaN}
  num_rows_processed | 7
  num_rows_skipped   | 0
  num_iterations     | 100
  -[ RECORD 2 ]------+-----------------------------------------------------------------------
  category           | 603
  coef               | {-572252710812.094,4826990013.37584,41195201231745.7,48061068103698.5}
  log_likelihood     | NaN
  std_err            | {NaN,NaN,NaN,NaN}
  z_stats            | {NaN,NaN,NaN,NaN}
  p_values           | {NaN,NaN,NaN,NaN}
  num_rows_processed | 7
  num_rows_skipped   | 0
  num_iterations     | 100



Thank you,
Atsushi



From: Frank McQuillan [mailto:fmcquillan@pivotal.io] 
Sent: Thursday, April 6, 2017 2:21 AM
To: user@madlib.incubator.apache.org
Subject: Re: Multinomial Regression: Failed with a msg "Hessian or gradient is not finite."

Atsushi-san,

The error that you see makes me think that you lost connection to the database in the middle
of the query:

"server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
The connection to the server was lost. Attempting reset: Failed."

I ran the same query with MADlib 1.10 on PG 9.4 and it seemed to work fine for me:

madlib=# SELECT * FROM mactbl_mini;
 ap1 | ap2 | ap3 | floor | id 
-----+-----+-----+-------+----
 -90 | -86 |   0 |   601 |  1
 -84 |   0 |   0 |   601 |  2
 -83 |   0 |   0 |   601 |  3
   0 | -72 | -84 |   601 |  6
   0 |   0 | -89 |   602 |  7
   0 |   0 |   0 |   602 |  8
   0 | -85 |   0 |   603 | 43
(7 rows)


madlib=# SELECT madlib.multinom('mactbl_mini',
madlib(#                      'mactbl_output',
madlib(#                       'floor',
madlib(#                       'ARRAY[1,
madlib'#                        ap1,
madlib'#                       ap2,
madlib'#                        ap3]',
madlib(#                        '601',
madlib(#                     'logit');
 multinom 
----------
 
(1 row)


madlib=# \x on
Expanded display is on.
madlib=# SELECT * FROM mactbl_output;
-[ RECORD 1 ]------+--------------------------------------------------------------------------------------
category           | 602
coef               | {112.327782768804,2.47679877229691,1.69069062458081,0.841889190837338}
log_likelihood     | NaN
std_err            | {5.81382676773028e+16,8.77427991785089e+17,1.06802178555571e+15,1.67358709762743e+15}
z_stats            | {1.93207997514273e-15,2.82279434379335e-18,1.58301136497989e-15,5.0304474265537e-16}
p_values           | {0.999999999999998,1,0.999999999999999,1}
num_rows_processed | 7
num_rows_skipped   | 0
num_iterations     | 100
-[ RECORD 2 ]------+--------------------------------------------------------------------------------------
category           | 603
coef               | {32.633774536969,1.61185753977669,-0.177075877885652,1.8769521118073}
log_likelihood     | NaN
std_err            | {9.40692076915866e+16,5.45753786923624e+18,1.30468621462021e+15,1.1421554354824e+22}
z_stats            | {3.46912399262056e-16,2.95345186491992e-19,-1.35722962273498e-16,1.6433420999433e-22}
p_values           | {1,1,1,1}
num_rows_processed | 7
num_rows_skipped   | 0
num_iterations     | 100



So please try again.

Frank

On Tue, Apr 4, 2017 at 7:28 PM, Neki, Atsushi <neki.atsushi@jp.fujitsu.com> wrote:
Hello,


I detected a failure of computation when using madlib.multinom function.
Is there anything wrong with my procedure?

====================================

- Issue description

madlib.multinom failed with the following message:

  > WARNING:  Hessian or gradient is not finite.
  > server closed the connection unexpectedly
  >     This probably means the server terminated abnormally
  >     before or while processing the request.
  > The connection to the server was lost. Attempting reset: Failed.


- Library

This issue happens in the following environment:
 1. madlib 1.9.1 + PostgreSQL 9.5
 2. madlib 1.10  + PostgreSQL 9.6


- Expectation

The result should be:

 1. Success computing the model
 2. Success building a model that can predict output precisely.
    (Because this is quite simple use case)


- Procedure to reproduce this issue

1. Prepare the following table:

testdb=# select * from mactbl_mini;
 ap1 | ap2 | ap3 | floor | id
-----+-----+-----+-------+----
 -90 | -86 |   0 | 601   |  1
 -84 |   0 |   0 | 601   |  2
 -83 |   0 |   0 | 601   |  3
   0 | -72 | -84 | 601   |  6
   0 |   0 | -89 | 602   |  7
   0 |   0 |   0 | 602   |  8
   0 | -85 |   0 | 603   | 43
(7 rows)


2. Compute a model to predict floor with ap by using madlib.multinom

testdb=# drop table mactbl_output;
DROP TABLE
testdb=# drop table mactbl_output_summary;
DROP TABLE
testdb=#
testdb=# SELECT madlib.multinom('mactbl_mini',
testdb(#                       'mactbl_output',
testdb(#                       'floor',
testdb(#                       'ARRAY[1,
testdb'#                       ap1,
testdb'#                       ap2,
testdb'#                       ap3]',
testdb(#                       '601',
testdb(#                       'logit');
WARNING:  Hessian or gradient is not finite.
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
!>


When I changed some parameter, it successed.

- Set '602' or '603' to ref_category.   [Success]
- Change max_iter from 100(default) to 10.       [Success]

====================================


Thanks in advance.

--
 Atsushi Neki

Mime
View raw message