[CouchDB] Issues with latest AWS image (Ver 3.2.2-8-r03 on Debian 11)

[Defcon089]

Platform

AWS

bndiagnostic ID know more about bndiagnostic ID

edfd39c1-430a-ac07-e22c-82e36132683d

bndiagnostic output

✓ Resources: No issues found
? Connectivity: Found possible issues
✓ Processes: No issues found

bndiagnostic was not useful. Could you please tell us why?

The suggested guides did not address my issue

Describe your issue as much as you can

Brand new AWS CouchDB image deployment. The Fauxton 'Verify CouchDB installation' fails at 'Create view' accompanied my massess of repeated error messages in the CouchDB logs.

Adding user fails and unable to add documents to any database that contains a _design/_auth with a "validate_doc_update" function. The same error message is seen in the error logs as when verifying installation.

[notice] 2022-08-15T05:53:21.286369Z couchdb@127.0.0.1 <0.5992.0> 4320b77450 localhost:15984 127.0.0.1 admin PUT /verifytestdb/test_doc_20 201 ok 12
[notice] 2022-08-15T05:53:21.302431Z couchdb@127.0.0.1 <0.5891.0> d77a9227e1 localhost:15984 127.0.0.1 admin PUT /verifytestdb/test_doc_30 201 ok 28
[notice] 2022-08-15T05:53:21.302589Z couchdb@127.0.0.1 <0.5995.0> c84e1b1300 localhost:15984 127.0.0.1 admin PUT /verifytestdb/test_doc_10 201 ok 28
[info] 2022-08-15T05:53:22.071451Z couchdb@127.0.0.1 <0.271.0> -------- couch_proc_manager <0.6418.0> died normal
[error] 2022-08-15T05:53:22.071701Z couchdb@127.0.0.1 <0.6411.0> -------- OS Process Error <0.6418.0> :: {os_process_error,{exit_status,139}}
[info] 2022-08-15T05:53:22.095045Z couchdb@127.0.0.1 <0.271.0> -------- couch_proc_manager <0.6461.0> died normal
[error] 2022-08-15T05:53:22.095222Z couchdb@127.0.0.1 <0.6411.0> -------- OS Process Error <0.6461.0> :: {os_process_error,{exit_status,139}}
[info] 2022-08-15T05:53:22.116691Z couchdb@127.0.0.1 <0.271.0> -------- couch_proc_manager <0.6465.0> died normal
[error] 2022-08-15T05:53:22.116859Z couchdb@127.0.0.1 <0.6411.0> -------- OS Process Error <0.6465.0> :: {os_process_error,{exit_status,139}}
...

[gongomgra]

Hi @Defcon089,

Thanks for using Bitnami. According to the logs you shared, it looks like some requests worked on your installation, so I understand your issue may be more related with CouchDB itself than with the Bitnami installation. We recommend you to open a new question in the CouchDB official support channels, where other people with more experience using the application may help you.

[Defcon089]

Hi @gongomgra
The issue is that this was a vanilla CouchDB deployment to an AWS EC2 using the Bitnami distributed image that is broken out of the box. We deployed multiple EC2s all with the same problem. There is no way to deploy a previous version Bitnami image on AWS so the only Bitnami CouchDB distribution available on the marketplace has these issues.

[gongomgra]

Hi @Defcon089,

Thanks for your message. Can you give us more information on how to reproduce your issue? Do you have any link to the guide you are following? Apart from that, can you also open a ticket in the official CouchDB forums? I think someone with more experience using CouchDB may be of help in this case (at least to help debugging the errors).

[Defcon089]

Hi @gongomgra
I launched a new CouchDB packaged by Bitnami instance in AWS as per the documentation taken directly from the Bitnami pages. I launched a t2.medium EC2 which is larger than the minimum Bitnami recommended t2.small EC2.
https://bitnami.com/stack/couchdb/cloud/aws
Once the EC2 had started I then forwarded port 5984 from the EC2 instance to my local workstation (you can also open up port 5984 from the AWS firewall and access directly) to access the CouchDB Fauxton web interface.
http://127.0.0.1:5984/_utils/#verifyinstall
The IP address and port will obviously vary with your individual circumstances.
I then ran the verify CouchDB installation as per the documentation
https://couchdb.apache.org/fauxton-visual-guide/index.html#verifyinstall
Note: you will need to authenticate to CouchDB using the Bitnami provided credentials.
The verification then fails on the 'Create View' step.
This appears an issue with the CouchDB build on the Bitnami distribution as we have seen no issues with CouchDB builds elsewhere.
Seeing a new Bitnami distribution was out (Ver 3.2.2-9-r04 on Debian 11) I launched a new instance with that hoping whatever issues there were with the distribution had been fixed but am seeing the same problems.

[Defcon089]

Hi @gongomgra
Some futher information...
I was not aware of the ability to find the past Bitnami images in the AWS community catalog. I have gone back and tested the previous Bitnami CouchDB images and have found the issue started with version 3.2.2-7-r02 on Debian 11 which was released 1-Aug-2022. The version previous to this ran through the CouchDB verify installation without any issues.
Further, the issues I have seen with the Bitnami distribution are identical to the issues described here
#apache/couchdb#4047
These issues are related to the Mac-App distribution of CouchDB though so not sure how that relates to the Bitnami distribution on the AWS marketplace. I found this link after noticing the following errors in /var/log/messages

Aug 25 06:19:40 ip-172-31-41-238 kernel: [  474.045115] couchjs[71575]: segfault at 0 ip 0000000000000000 sp 00007fff2ce5d5b8 error 14 in couchjs[556280f4a000+3000]
Aug 25 06:19:40 ip-172-31-41-238 kernel: [  474.058731] Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.
Aug 25 06:19:40 ip-172-31-41-238 kernel: [  474.082035] couchjs[71578]: segfault at 0 ip 0000000000000000 sp 00007ffcc6eb6f48 error 14 in couchjs[55b04a892000+3000]
Aug 25 06:19:40 ip-172-31-41-238 kernel: [  474.095369] Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.

[mdhont]

It seems to be a bug of the application itself. I recommend asking in the official forum.

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[Defcon089]

@mdhont the bug you refer to is from over 4 years ago. The Bitnami releases prior to 1st August 2022 as detailed in my previous post were working fine so this hasn't been an issue.
It is beyond disappointing to see that there has been yet another release (version 3.2.2-10-r05) on the AWS marketplace of 'CouchDB packaged by Bitnami' distributed by 'Bitnami by VMware' that is broken with the same issues described above.
I was hoping that by posting here that some feedback would filter through to whoever is releasing these images to the AWS marketeplace but unfortunately that does not appear to be the case.

[gongomgra]

Hi @Defcon089 ,

Thanks for your message and sorry for the delay on getting back to you. I reproduced your issue but I couldn't find the reason causing this (still checking). From the logs, I also found the command causing the segmentation fault

[error] 2022-09-30T10:04:46.129515Z couchdb@127.0.0.1 <0.3717.13> -------- gen_server <0.3717.13> terminated with reason: {exit_status,139}
  last msg: redacted
     state: {os_proc,"./bin/couchjs ./share/server/main.js",#Port<0.281853>,fun couch_os_process:writejson/2,fun couch_os_process:readjson/1,5000,300000}
    extra: []
[error] 2022-09-30T10:04:46.129579Z couchdb@127.0.0.1 <0.3717.13> -------- gen_server <0.3717.13> terminated with reason: {exit_status,139}
  last msg: redacted
     state: {os_proc,"./bin/couchjs ./share/server/main.js",#Port<0.281853>,fun couch_os_process:writejson/2,fun couch_os_process:readjson/1,5000,300000}
    extra: []
[error] 2022-09-30T10:04:46.129675Z couchdb@127.0.0.1 <0.3717.13> -------- CRASH REPORT Process  (<0.3717.13>) with 0 neighbors exited with reason: {exit_status,139} at gen_server:handle_common_reply/8(line:811) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couc
h_os_process,init,['Argument__1']}, ancestors: [<0.3716.13>], message_queue_len: 0, links: [], dictionary: [], trap_exit: false, status: running, heap_size: 987, stack_size: 29, reductions: 2372
[error] 2022-09-30T10:04:46.129774Z couchdb@127.0.0.1 <0.3717.13> -------- CRASH REPORT Process  (<0.3717.13>) with 0 neighbors exited with reason: {exit_status,139} at gen_server:handle_common_reply/8(line:811) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couc
h_os_process,init,['Argument__1']}, ancestors: [<0.3716.13>], message_queue_len: 0, links: [], dictionary: [], trap_exit: false, status: running, heap_size: 987, stack_size: 29, reductions: 2372

And it is true it generates a segmentation fault error running it from the command line. I'm trying to find out more information about this. I will keep you posted.

[gongomgra]

I found our couchjs binary is stripped. I will generate a new one including the debugging symbols and give it another try.

$ file /opt/bitnami/couchdb/bin/couchjs
/opt/bitnami/couchdb/bin/couchjs: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=5b3a6e471d602b16a9fbb671bf8c1357e43f8334, for GNU/Linux 3.2.0, stripped

[gongomgra]

Hi @Defcon089,

I built both the couchjs binary and the libmoz library (from SpiderMonkey) with debugging symbols and took a look to the generated core dumped. However, the backtrace command didn't provide much information

$ file /opt/bitnami/couchdb/bin/couchjs
/opt/bitnami/couchdb/bin/couchjs: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=5b3a6e471d602b16a9fbb671bf8c1357e43f8334, for GNU/Linux 3.2.0, with debug_info, not stripped

$ file /opt/bitnami/common/lib/libmozjs-91.so
/opt/bitnami/common/lib/libmozjs-91.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=99a776915c3495a0bf3aae8eda6cfec1f9aa3ca6, with debug_info, not stripped

$ gdb /opt/bitnami/couchdb/bin/couchjs /opt/bitnami/couchdb/core
GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /opt/bitnami/couchdb/bin/couchjs...
[New LWP 3348]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/opt/bitnami/couchdb/bin/couchjs /opt/bitnami/couchdb/share/server/main.js'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007facabb2b8b0 in js::MutexImpl::MutexImpl (this=0x7facad76ea00 <js::gHelperThreadLock>) at /root/firefox-91.13.0/js/src/threading/Mutex.h:39
#2  js::Mutex::Mutex (id=..., this=0x7facad76ea00 <js::gHelperThreadLock>) at /root/firefox-91.13.0/js/src/threading/Mutex.h:73
#3  __static_initialization_and_destruction_0 (__priority=65535, __initialize_p=1) at /root/firefox-91.13.0/js/src/vm/HelperThreads.cpp:61
#4  _GLOBAL__sub_I_Unified_cpp_js_src20.cpp(void) () at /root/firefox-91.13.0/js/src/vm/InternalThreadPool.cpp:278
#5  0x00007facad785fe2 in call_init (l=<optimized out>, argc=argc@entry=2, argv=argv@entry=0x7ffdc7dbbb78, env=env@entry=0x7ffdc7dbbb90) at dl-init.c:72
#6  0x00007facad7860e9 in call_init (env=0x7ffdc7dbbb90, argv=0x7ffdc7dbbb78, argc=2, l=<optimized out>) at dl-init.c:30
#7  _dl_init (main_map=0x7facad7a2180, argc=2, argv=0x7ffdc7dbbb78, env=0x7ffdc7dbbb90) at dl-init.c:119
#8  0x00007facad7770ca in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#9  0x0000000000000002 in ?? ()
#10 0x00007ffdc7dbc2d8 in ?? ()
#11 0x00007ffdc7dbc2f9 in ?? ()
#12 0x0000000000000000 in ?? ()
(gdb) exit

Apart from that, I launched the official Docker image for CouchDB 3.2.2, and I realized they are using another version of SpiderMonkey.

$ docker run --rm -it -u root couchdb bash
# /opt/couchdb/bin/couchjs -V
couchjs - Apache CouchDB 3.2.2 (SpiderMonkey 86)

Licensed under the Apache License, Version 2.0 (the "License"); you may not use
this file except in compliance with the License. You may obtain a copy ofthe
License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.

# ldd /opt/couchdb/bin/couchjs
        linux-vdso.so.1 (0x00007ffd2bbfe000)
        libmozjs-78.so.0 => /usr/lib/x86_64-linux-gnu/libmozjs-78.so.0 (0x00007ff67f67f000)

I also verify we were using a different SpiderMonkey version in the previous image (the one you mentioned that worked fine). I will open a new task for our engineering team to review this and check if using another version of SpiderMonkey solves the issue.

Unfortunately, we can't provide you with any estimation for this to be solved, but we will post any update in this thread.

[gongomgra]

Hi @Defcon089, @AnalogJ,

Just a quick note to let you know that we have found a fix for this issue and that we are in the process of releasing a new revision of our CouchDB 3.x assets including the fix. It was caused by the way we were building the SpiderMonkey library.

I'm also sharing a screenshot of the updated solution successfully running the "verify installation" process

We will post an update once the different assets are released.

[gongomgra]

Hi @Defcon089,

We have released all assets related to CouchDB including the fix. You can download the latest revision of them, or get the new AWS AMI IDs from our website

https://bitnami.com/stack/couchdb