SEGV in sampling

[jpritikin]

Summary:

SEGV in sampling

Description:

Program received signal SIGSEGV, Segmentation fault.
0x000055555556c003 in stan::math::(anonymous namespace)::multiply_vv_vari::chain() ()
#0 0x000055555556c003 in stan::math::(anonymous namespace)::multiply_vv_vari::chain() ()
#1 0x00005555555d3a22 in void stan::math::gradient<stan::model::model_functional<model_fp_model_namespace::model_fp_model> >(stan::model::model_functional<model_fp_model_namespace::model_fp_model> const&, Eigen::Matrix<double, -1, 1, 0, -1, 1> const&, double&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&) ()
#2 0x00005555555d3cb3 in void stan::model::gradient<model_fp_model_namespace::model_fp_model>(model_fp_model_namespace::model_fp_model const&, Eigen::Matrix<double, -1, 1, 0, -1, 1> const&, double&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, stan::callbacks::logger&) ()
#3 0x00005555555d5755 in stan::mcmc::base_hamiltonian<model_fp_model_namespace::model_fp_model, stan::mcmc::diag_e_point, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::update_potential_gradient(stan::mcmc::diag_e_point&, stan::callbacks::logger&) ()
#4 0x00005555555d5f74 in stan::mcmc::base_leapfrog<stan::mcmc::diag_e_metric<model_fp_model_namespace::model_fp_model, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > > >::evolve(stan::mcmc::diag_e_point&, stan::mcmc::diag_e_metric<model_fp_model_namespace::model_fp_model, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >&, double, stan::callbacks::logger&) ()
#5 0x00005555555f2ee7 in stan::mcmc::base_nuts<model_fp_model_namespace::model_fp_model, stan::mcmc::diag_e_metric, stan::mcmc::expl_leapfrog, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::build_tree(int, stan::mcmc::ps_point&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, double, double, int&, double&, double&, stan::callbacks::logger&) ()
#6 0x00005555555f2dae in stan::mcmc::base_nuts<model_fp_model_namespace::model_fp_model, stan::mcmc::diag_e_metric, stan::mcmc::expl_leapfrog, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::build_tree(int, stan::mcmc::ps_point&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, double, double, int&, double&, double&, stan::callbacks::logger&) ()

Reproducible Steps:

It happens very rarely. Try running the attached model & data for a long time.
model-fp.stan.txt
s08.data.R.txt

I used gdb ./model-fp -n -batch -x ./s08.gdb where s08.gdb is

  run sample num_samples=1000 num_warmup=1000      algorithm=hmc engine=nuts max_depth=15      data file=s08.data.R      output file=data/s08_1.csv
  where

Current Version:

v2.17.1
gcc 8.1

[bob-carpenter]

When you say a long time, what scale are you talking about? What kind of hardware? And is this only a problem in gdb or does it also happen when you run without debugging, too?

Thanks for capturing the traceback. The only thing I can imagine would cause a problem in the chain() method is an issue with pointer chasing on the autodiff stack.

This kind of thing can be caused by hardware, but I wouldn't want to jump to that conclusion too quickly. I'll try to reproduce first.

[jpritikin]

When you say a long time, what scale are you talking about?

It happened once during 2000 samples per 6 chains. The traceback looked similar to the SEGVs I encountered with gcc 7 so I figured it was probably a real bug. I'll try running it again to see whether it re-occurs.

What kind of hardware?

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 142
model name : Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
stepping : 10
microcode : 0x84
cpu MHz : 2923.442
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 22
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves ibpb ibrs stibp dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips : 3984.00
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

And is this only a problem in gdb or does it also happen when you run without debugging, too?

I haven't tried without gdb.

This kind of thing can be caused by hardware

Maybe, but the laptop is less than 6 months old.

[jpritikin]

I got "lucky" again:

Program received signal SIGSEGV, Segmentation fault.
0x00005555555d3a1f in void stan::math::gradient<stan::model::model_functional<model_fp_model_namespace::model_fp_model> >(stan::model::model_functional<model_fp_model_namespace::model_fp_model> const&, Eigen::Matrix<double, -1, 1, 0, -1, 1> const&, double&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&) ()
#0  0x00005555555d3a1f in void stan::math::gradient<stan::model::model_functional<model_fp_model_namespace::model_fp_model> >(stan::model::model_functional<model_fp_model_namespace::model_fp_model> const&, Eigen::Matrix<double, -1, 1, 0, -1, 1> const&, double&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&) ()
#1  0x00005555555d3cb3 in void stan::model::gradient<model_fp_model_namespace::model_fp_model>(model_fp_model_namespace::model_fp_model const&, Eigen::Matrix<double, -1, 1, 0, -1, 1> const&, double&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, stan::callbacks::logger&) ()
#2  0x00005555555d5755 in stan::mcmc::base_hamiltonian<model_fp_model_namespace::model_fp_model, stan::mcmc::diag_e_point, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::update_potential_gradient(stan::mcmc::diag_e_point&, stan::callbacks::logger&) ()
#3  0x00005555555d5f74 in stan::mcmc::base_leapfrog<stan::mcmc::diag_e_metric<model_fp_model_namespace::model_fp_model, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > > >::evolve(stan::mcmc::diag_e_point&, stan::mcmc::diag_e_metric<model_fp_model_namespace::model_fp_model, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >&, double, stan::callbacks::logger&) ()
#4  0x00005555555f2ee7 in stan::mcmc::base_nuts<model_fp_model_namespace::model_fp_model, stan::mcmc::diag_e_metric, stan::mcmc::expl_leapfrog, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::build_tree(int, stan::mcmc::ps_point&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, double, double, int&, double&, double&, stan::callbacks::logger&) ()
#5  0x00005555555f2dae in stan::mcmc::base_nuts<model_fp_model_namespace::model_fp_model, stan::mcmc::diag_e_metric, stan::mcmc::expl_leapfrog, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::build_tree(int, stan::mcmc::ps_point&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, double, double, int&, double&, double&, stan::callbacks::logger&) ()
#6  0x00005555555f2dae in stan::mcmc::base_nuts<model_fp_model_namespace::model_fp_model, stan::mcmc::diag_e_metric, stan::mcmc::expl_leapfrog, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::build_tree(int, stan::mcmc::ps_point&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, double, double, int&, double&, double&, stan::callbacks::logger&) ()
#7  0x00005555555f2e97 in stan::mcmc::base_nuts<model_fp_model_namespace::model_fp_model, stan::mcmc::diag_e_metric, stan::mcmc::expl_leapfrog, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::build_tree(int, stan::mcmc::ps_point&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, double, double, int&, double&, double&, stan::callbacks::logger&) ()
#8  0x00005555555f2e97 in stan::mcmc::base_nuts<model_fp_model_namespace::model_fp_model, stan::mcmc::diag_e_metric, stan::mcmc::expl_leapfrog, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::build_tree(int, stan::mcmc::ps_point&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, double, double, int&, double&, double&, stan::callbacks::logger&) ()
#9  0x00005555555f2dae in stan::mcmc::base_nuts<model_fp_model_namespace::model_fp_model, stan::mcmc::diag_e_metric, stan::mcmc::expl_leapfrog, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::build_tree(int, stan::mcmc::ps_point&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, double, double, int&, double&, double&, stan::callbacks::logger&) ()
#10 0x00005555555f2dae in stan::mcmc::base_nuts<model_fp_model_namespace::model_fp_model, stan::mcmc::diag_e_metric, stan::mcmc::expl_leapfrog, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::build_tree(int, stan::mcmc::ps_point&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, double, double, int&, double&, double&, stan::callbacks::logger&) ()
#11 0x00005555555f3a33 in stan::mcmc::base_nuts<model_fp_model_namespace::model_fp_model, stan::mcmc::diag_e_metric, stan::mcmc::expl_leapfrog, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::transition(stan::mcmc::sample&, stan::callbacks::logger&) ()
#12 0x00005555555f3f5c in stan::mcmc::adapt_diag_e_nuts<model_fp_model_namespace::model_fp_model, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::transition(stan::mcmc::sample&, stan::callbacks::logger&) ()
#13 0x00005555555c1c93 in void stan::services::util::generate_transitions<model_fp_model_namespace::model_fp_model, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >(stan::mcmc::base_mcmc&, int, int, int, int, int, bool, bool, stan::services::util::mcmc_writer&, stan::mcmc::sample&, model_fp_model_namespace::model_fp_model&, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> >&, stan::callbacks::interrupt&, stan::callbacks::logger&) ()
#14 0x00005555555f086d in void stan::services::util::run_adaptive_sampler<stan::mcmc::adapt_diag_e_nuts<model_fp_model_namespace::model_fp_model, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >, model_fp_model_namespace::model_fp_model, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >(stan::mcmc::adapt_diag_e_nuts<model_fp_model_namespace::model_fp_model, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >&, model_fp_model_namespace::model_fp_model&, std::vector<double, std::allocator<double> >&, int, int, int, int, bool, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> >&, stan::callbacks::interrupt&, stan::callbacks::logger&, stan::callbacks::writer&, stan::callbacks::writer&) ()
#15 0x00005555555f1228 in int stan::services::sample::hmc_nuts_diag_e_adapt<model_fp_model_namespace::model_fp_model>(model_fp_model_namespace::model_fp_model&, stan::io::var_context&, stan::io::var_context&, unsigned int, unsigned int, double, int, int, int, bool, int, double, double, int, double, double, double, double, unsigned int, unsigned int, unsigned int, stan::callbacks::interrupt&, stan::callbacks::logger&, stan::callbacks::writer&, stan::callbacks::writer&, stan::callbacks::writer&) ()
#16 0x00005555555ff3eb in int cmdstan::command<model_fp_model_namespace::model_fp_model>(int, char const**) ()
#17 0x000055555556a5a6 in main ()

And again:

Program received signal SIGSEGV, Segmentation fault.
0x00005555555d3a1f in void stan::math::gradient<stan::model::model_functional<model_fp_model_namespace::model_fp_model> >(stan::model::model_functional<model_fp_model_namespace::model_fp_model> const&, Eigen::Matrix<double, -1, 1, 0, -1, 1> const&, double&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&) ()
#0  0x00005555555d3a1f in void stan::math::gradient<stan::model::model_functional<model_fp_model_namespace::model_fp_model> >(stan::model::model_functional<model_fp_model_namespace::model_fp_model> const&, Eigen::Matrix<double, -1, 1, 0, -1, 1> const&, double&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&) ()
#1  0x00005555555d3cb3 in void stan::model::gradient<model_fp_model_namespace::model_fp_model>(model_fp_model_namespace::model_fp_model const&, Eigen::Matrix<double, -1, 1, 0, -1, 1> const&, double&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, stan::callbacks::logger&) ()
#2  0x00005555555d5755 in stan::mcmc::base_hamiltonian<model_fp_model_namespace::model_fp_model, stan::mcmc::diag_e_point, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::update_potential_gradient(stan::mcmc::diag_e_point&, stan::callbacks::logger&) ()
#3  0x00005555555d5f74 in stan::mcmc::base_leapfrog<stan::mcmc::diag_e_metric<model_fp_model_namespace::model_fp_model, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > > >::evolve(stan::mcmc::diag_e_point&, stan::mcmc::diag_e_metric<model_fp_model_namespace::model_fp_model, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >&, double, stan::callbacks::logger&) ()
#4  0x00005555555f2ee7 in stan::mcmc::base_nuts<model_fp_model_namespace::model_fp_model, stan::mcmc::diag_e_metric, stan::mcmc::expl_leapfrog, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::build_tree(int, stan::mcmc::ps_point&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, double, double, int&, double&, double&, stan::callbacks::logger&) ()
#5  0x00005555555f2e97 in stan::mcmc::base_nuts<model_fp_model_namespace::model_fp_model, stan::mcmc::diag_e_metric, stan::mcmc::expl_leapfrog, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::build_tree(int, stan::mcmc::ps_point&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, double, double, int&, double&, double&, stan::callbacks::logger&) ()
#6  0x00005555555f2e97 in stan::mcmc::base_nuts<model_fp_model_namespace::model_fp_model, stan::mcmc::diag_e_metric, stan::mcmc::expl_leapfrog, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::build_tree(int, stan::mcmc::ps_point&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, double, double, int&, double&, double&, stan::callbacks::logger&) ()
#7  0x00005555555f2dae in stan::mcmc::base_nuts<model_fp_model_namespace::model_fp_model, stan::mcmc::diag_e_metric, stan::mcmc::expl_leapfrog, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::build_tree(int, stan::mcmc::ps_point&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, double, double, int&, double&, double&, stan::callbacks::logger&) ()
#8  0x00005555555f2dae in stan::mcmc::base_nuts<model_fp_model_namespace::model_fp_model, stan::mcmc::diag_e_metric, stan::mcmc::expl_leapfrog, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::build_tree(int, stan::mcmc::ps_point&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, double, double, int&, double&, double&, stan::callbacks::logger&) ()
#9  0x00005555555f2e97 in stan::mcmc::base_nuts<model_fp_model_namespace::model_fp_model, stan::mcmc::diag_e_metric, stan::mcmc::expl_leapfrog, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::build_tree(int, stan::mcmc::ps_point&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, double, double, int&, double&, double&, stan::callbacks::logger&) ()
#10 0x00005555555f2e97 in stan::mcmc::base_nuts<model_fp_model_namespace::model_fp_model, stan::mcmc::diag_e_metric, stan::mcmc::expl_leapfrog, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::build_tree(int, stan::mcmc::ps_point&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, double, double, int&, double&, double&, stan::callbacks::logger&) ()
#11 0x00005555555f2e97 in stan::mcmc::base_nuts<model_fp_model_namespace::model_fp_model, stan::mcmc::diag_e_metric, stan::mcmc::expl_leapfrog, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::build_tree(int, stan::mcmc::ps_point&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&, double, double, int&, double&, double&, stan::callbacks::logger&) ()
#12 0x00005555555f3cf7 in stan::mcmc::base_nuts<model_fp_model_namespace::model_fp_model, stan::mcmc::diag_e_metric, stan::mcmc::expl_leapfrog, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::transition(stan::mcmc::sample&, stan::callbacks::logger&) ()
#13 0x00005555555f3f5c in stan::mcmc::adapt_diag_e_nuts<model_fp_model_namespace::model_fp_model, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >::transition(stan::mcmc::sample&, stan::callbacks::logger&) ()
#14 0x00005555555c1c93 in void stan::services::util::generate_transitions<model_fp_model_namespace::model_fp_model, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >(stan::mcmc::base_mcmc&, int, int, int, int, int, bool, bool, stan::services::util::mcmc_writer&, stan::mcmc::sample&, model_fp_model_namespace::model_fp_model&, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> >&, stan::callbacks::interrupt&, stan::callbacks::logger&) ()
#15 0x00005555555f086d in void stan::services::util::run_adaptive_sampler<stan::mcmc::adapt_diag_e_nuts<model_fp_model_namespace::model_fp_model, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >, model_fp_model_namespace::model_fp_model, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >(stan::mcmc::adapt_diag_e_nuts<model_fp_model_namespace::model_fp_model, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> > >&, model_fp_model_namespace::model_fp_model&, std::vector<double, std::allocator<double> >&, int, int, int, int, bool, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014u, 0u, 2147483563u>, boost::random::linear_congruential_engine<unsigned int, 40692u, 0u, 2147483399u> >&, stan::callbacks::interrupt&, stan::callbacks::logger&, stan::callbacks::writer&, stan::callbacks::writer&) ()
#16 0x00005555555f1228 in int stan::services::sample::hmc_nuts_diag_e_adapt<model_fp_model_namespace::model_fp_model>(model_fp_model_namespace::model_fp_model&, stan::io::var_context&, stan::io::var_context&, unsigned int, unsigned int, double, int, int, int, bool, int, double, double, int, double, double, double, double, unsigned int, unsigned int, unsigned int, stan::callbacks::interrupt&, stan::callbacks::logger&, stan::callbacks::writer&, stan::callbacks::writer&, stan::callbacks::writer&) ()
#17 0x00005555555ff3eb in int cmdstan::command<model_fp_model_namespace::model_fp_model>(int, char const**) ()
#18 0x000055555556a5a6 in main ()

These correspond to seeds 3468399826 and 3468399722.

[jpritikin]

I re-ran with seed set to 3468399826 and sampling completed without difficulty. There must be some other source of indeterminacy.

[bob-carpenter]

The same seed should lead to exactly the same results if nothing else changes (including hardware, OS, compiler, compiler settings, etc.).

[jpritikin]

OK, I'm stumped. Maybe these SEGVs are caused by cosmic rays. I didn't expect the probability of random memory corruption was so high. I observed these two SEGVs on the same day. How do you get regression tests to pass if computers involve so much random noise? Maybe ECC memory is required?

[jpritikin]

On a 2012 laptop, I ran memtest86+ for 12 hours with no errors detected. Maybe my new laptop is not as robust to cosmic rays?

[bob-carpenter]

As soon as I started hammering machines with numerical jobs 15 or so years ago, I needed to switch to ECC memory. Having said that, I've never had a problem on my Macbook Pro other than with Safari crashing the whole OS. I have no idea what the robustness is like now for other platforms or how much it varies board-to-board and chip-to-chip. Intermittent failures are known to be the hardest thing to debug. The best programmer I've ever known was a genius at finding race conditions in telephony platforms that took 48 hours to arise under 100s of simultaneous threads. We do have problems with regression tests running end-to-end. Mainly network interrupts and things like that throwing wrenches into the works that are beyond the continuous integration frameworks' abilities to self clean. Not with random segfaults.

[jpritikin]

The best programmer I've ever known was a genius at finding race conditions in telephony platforms that took 48 hours to arise under 100s of simultaneous threads.

Definitely things get much more complex with multiple threads, but a single Stan sampler only has 1 thread!

Not with random segfaults.

Yeah, I am surprised that the symptom is consistently a SEGV. My experience is that SEGVs usually indicate a real problem that is not difficult to reproduce. I'm really puzzled with the current issue.

[bob-carpenter]

That has not been my experience. If they're something that happens from overwriting memory accidentally, it can take a long time until something gets corrupted in just the right way to be read as an adress rather than as data.

[jpritikin]

I ran the model in valgrind overnight. No errors.

[bob-carpenter]

Thanks for trying to track that down and I'm glad to hear it passed that test. It's not definitive, though, as errors can be masked by profilers, especially if they come up very intermittently as race conditions. Let's hope it's not that, as it's very hard to track down.

[jpritikin]

I don't know if it's related, but the laptop I was using did have some hardware issue(s). It was under warranty and I sent it in for repairs.

[bob-carpenter]

Thanks for the follow up. I'm going to go ahead and close this issue until it's reproducible.