madlib-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nandish Jayaram <njaya...@pivotal.io>
Subject Re: Out of memory for neural network
Date Fri, 27 Jul 2018 16:59:32 GMT
Thank you Luyao. I have created
https://issues.apache.org/jira/browse/MADLIB-1259 to track this issue.

NJ

On Fri, Jul 27, 2018 at 8:51 AM LUYAO CHEN <luyao_chen@hotmail.com> wrote:

> Hi NJ,
>
>
> I use the same date set for the issue
> https://issues.apache.org/jira/browse/MADLIB-1257.
>
>
> It has 23K instances / 300 features in 263 groups.
>
> SELECT madlib.mlp_classification(
>     'train_data',      -- Source table
>     'mlp_model',      -- Destination table
>     'features',     -- Input features
>     'positive',     -- Label
>     ARRAY[5],         -- Number of units per layer
>     'learning_rate_init=0.003,
>     n_iterations=500,
>     tolerance=0',     -- Optimizer params
>     'tanh',           -- Activation function
>     NULL,             -- Default weight (1)
>     FALSE,            -- No warm start
>     true,             -- verbose
>     'caseid'         -- Grouping
> );
>
> PostgreSQL memory setup.
>
>  checkpoint_completion_target = '0.9';
>  default_statistics_target = '500';
>  effective_cache_size = '12GB';
>  effective_io_concurrency = '200';
>  maintenance_work_mem = '2GB';
>  max_connections = '20';
>  max_parallel_workers = '4';
>  max_parallel_workers_per_gather = '2';
>  max_wal_size = '8GB';
>  max_worker_processes = '4';
>  min_wal_size = '4GB';
>  random_page_cost = '1.1';
>  shared_buffers = '4GB';
>  wal_buffers = '16MB';
>  work_mem = '52428kB';
>
> also i setup sysctl -w vm.overcommit_memory=2 to avoide the crash of
> postmaster
>
> Regards,
> Luyao
>
>
> ------------------------------
> *From:* Nandish Jayaram <njayaram@pivotal.io>
> *Sent:* Friday, July 27, 2018 10:05 AM
> *To:* user@madlib.apache.org
> *Subject:* Re: Out of memory for neural network
>
> Hi Luyao,
>
> Can you kindly include details about the database settings (btw, are you
> using Postgres or Greenplum?) and the training dataset size used?
> If your dataset is publicly available, please do share the details of
> where we can access it too.
>
> NJ
>
> Sent from my iPhone
>
> On Jul 27, 2018, at 7:41 AM, LUYAO CHEN <luyao_chen@hotmail.com> wrote:
>
> Dear user community,
>
>
> I would report a problem regarding in neural network. I am using a 16G
> RAM machine.
>
>
> After some iterations (~100 ). I got the below error
>
>
> ERROR:  spiexceptions.OutOfMemory: out of memory
> DETAIL:  Failed on request of size 32800.
> CONTEXT:  Traceback (most recent call last):
>   PL/Python function "mlp_classification", line 36, in <module>
>     grouping_col
>   PL/Python function "mlp_classification", line 45, in wrapper
>   PL/Python function "mlp_classification", line 325, in mlp
>   PL/Python function "mlp_classification", line 580, in update
> PL/Python function "mlp_classification"
>
> Below is the command ,
>
> SELECT madlib.mlp_classification(
>     'train_data_sub',      -- Source table
>     'mlp_model',      -- Destination table
>     'features',     -- Input features
>     'positive',     -- Label
>     ARRAY[5],         -- Number of units per layer
>     'learning_rate_init=0.003,
>     n_iterations=500,
>     tolerance=0',     -- Optimizer params
>     'tanh',           -- Activation function
>     NULL,             -- Default weight (1)
>     FALSE,            -- No warm start
>     true,             -- verbose
>     'case_icd'         -- Grouping
> );
>
> Is that a problem or just caused by the data size?
>
> Regards,
> Luyao Chen
>
>

Mime
View raw message