madlib-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From LUYAO CHEN <luyao_c...@hotmail.com>
Subject Re: Out of memory for neural network
Date Fri, 27 Jul 2018 15:51:05 GMT
Hi NJ,


I use the same date set for the issue https://issues.apache.org/jira/browse/MADLIB-1257.


It has 23K instances / 300 features in 263 groups.

SELECT madlib.mlp_classification(
    'train_data',      -- Source table
    'mlp_model',      -- Destination table
    'features',     -- Input features
    'positive',     -- Label
    ARRAY[5],         -- Number of units per layer
    'learning_rate_init=0.003,
    n_iterations=500,
    tolerance=0',     -- Optimizer params
    'tanh',           -- Activation function
    NULL,             -- Default weight (1)
    FALSE,            -- No warm start
    true,             -- verbose
    'caseid'         -- Grouping
);
PostgreSQL memory setup.

 checkpoint_completion_target = '0.9';
 default_statistics_target = '500';
 effective_cache_size = '12GB';
 effective_io_concurrency = '200';
 maintenance_work_mem = '2GB';
 max_connections = '20';
 max_parallel_workers = '4';
 max_parallel_workers_per_gather = '2';
 max_wal_size = '8GB';
 max_worker_processes = '4';
 min_wal_size = '4GB';
 random_page_cost = '1.1';
 shared_buffers = '4GB';
 wal_buffers = '16MB';
 work_mem = '52428kB';

also i setup sysctl -w vm.overcommit_memory=2 to avoide the crash of postmaster

Regards,
Luyao


________________________________
From: Nandish Jayaram <njayaram@pivotal.io>
Sent: Friday, July 27, 2018 10:05 AM
To: user@madlib.apache.org
Subject: Re: Out of memory for neural network

Hi Luyao,

Can you kindly include details about the database settings (btw, are you using Postgres or
Greenplum?) and the training dataset size used?
If your dataset is publicly available, please do share the details of where we can access
it too.

NJ

Sent from my iPhone

On Jul 27, 2018, at 7:41 AM, LUYAO CHEN <luyao_chen@hotmail.com<mailto:luyao_chen@hotmail.com>>
wrote:


Dear user community,


I would report a problem regarding in neural network. I am using a 16G RAM machine.


After some iterations (~100 ). I got the below error


ERROR:  spiexceptions.OutOfMemory: out of memory
DETAIL:  Failed on request of size 32800.
CONTEXT:  Traceback (most recent call last):
  PL/Python function "mlp_classification", line 36, in <module>
    grouping_col
  PL/Python function "mlp_classification", line 45, in wrapper
  PL/Python function "mlp_classification", line 325, in mlp
  PL/Python function "mlp_classification", line 580, in update
PL/Python function "mlp_classification"

Below is the command ,

SELECT madlib.mlp_classification(
    'train_data_sub',      -- Source table
    'mlp_model',      -- Destination table
    'features',     -- Input features
    'positive',     -- Label
    ARRAY[5],         -- Number of units per layer
    'learning_rate_init=0.003,
    n_iterations=500,
    tolerance=0',     -- Optimizer params
    'tanh',           -- Activation function
    NULL,             -- Default weight (1)
    FALSE,            -- No warm start
    true,             -- verbose
    'case_icd'         -- Grouping
);

Is that a problem or just caused by the data size?

Regards,
Luyao Chen


Mime
View raw message