Missing (node.js) Upstream when running example project

Issues during installation and related to database
Post Reply
Reza.Nazeman
Posts: 5
Joined: Tue Jun 12, 2018 3:59 pm

Missing (node.js) Upstream when running example project

Post by Reza.Nazeman » Tue Jun 12, 2018 4:41 pm

We have an issue running the example project. Clicking on "Create Document" shows only the loading page and end up in the following error after a while:
Loading.PNG
Loading.PNG (95.19 KiB) Viewed 558 times
Loading2.PNG
Loading2.PNG (22.39 KiB) Viewed 558 times
In the nginx logs we found a missing upstream:

Code: Select all

2018/06/12 17:57:16 [error] 1244#0: *9 no live upstreams while connecting to upstream, client: 10.250.52.51, server: , request: "GET /2018-05-22-13-44/doc/-1383711616/c/info?t=1528819037024 HTTP/1.1", upstream: "http://docservice/doc/-1383711616/c/info", host: "onlyoffice.local", referrer: "https://onlyoffice.local/2018-05-22-13-44/web-apps/apps/documenteditor/main/index.html?_dc=2018-05-22-13-44&lang=en&customer=ONLYOFFICE&frameEditorId=iframeEditor"
Checking the file /etc/nginx/includes/onlyoffice-http.conf we found the configured upstream:

Code: Select all

upstream docservice {
  server localhost:8000;
}
After some more research, we figured out that the node-js server should be serving port 8000 with the document-service, but it's not running, port 8000 is missing in the list:

Code: Select all

$ sudo netstat -ntlp | grep node
tcp6       0      0 :::8080                 :::*                    LISTEN      1393/node
tcp6       0      0 :::3000                 :::*                    LISTEN      1805/node
Digging into /var/log/onlyoffice/documentserver/docservice/out.log we find a bunch of these messages:

Code: Select all

[2018-06-12 17:51:07.990] [ERROR] nodeJS - [AMQP] Error: connect EMFILE 127.0.0.1:5672 - Local (undefined:undefined)
    at Object.exports._errnoException (util.js:1020:11)
    at exports._exceptionWithHostPort (util.js:1043:20)
    at connect (net.js:892:16)
    at net.js:1028:7
    at /var/www/onlyoffice/documentserver/server/Common/node_modules/dnscache/lib/index.js:80:28
    at /var/www/onlyoffice/documentserver/server/Common/node_modules/dnscache/lib/cache.js:116:13
    at RawTask.call (/var/www/onlyoffice/documentserver/server/Common/node_modules/asap/asap.js:40:19)
    at flush (/var/www/onlyoffice/documentserver/server/Common/node_modules/asap/raw.js:50:29)
    at _combinedTickCallback (internal/process/next_tick.js:73:7)
    at process._tickCallback (internal/process/next_tick.js:104:9)
The error message indicates, that the docserver is running out of file-descriptors.

The ulimits are pretty high, so checking the open files of nodeJs leads us to here:

Code: Select all

node      1390 onlyoffice   10u  IPv4   20221      0t0  TCP 127.0.0.1:43632->127.0.0.1:6379 (ESTABLISHED)
node      1391 onlyoffice   10u  IPv4   18982      0t0  TCP 127.0.0.1:43630->127.0.0.1:6379 (ESTABLISHED)
node      1393 onlyoffice   11u  IPv6   20206      0t0  TCP *:8080 (LISTEN)
node      1394 onlyoffice   11u  IPv4   21569      0t0  TCP 127.0.0.1:43658->127.0.0.1:6379 (ESTABLISHED)
node      1805 onlyoffice   10u  IPv6   58810      0t0  TCP *:3000 (LISTEN)
node      3340 onlyoffice   11u  IPv4 4305590      0t0  TCP 127.0.0.1:41246->127.0.0.1:6379 (ESTABLISHED)
node      3340 onlyoffice   12u  IPv4 7009241      0t0  TCP 127.0.0.1:54468->127.0.0.1:5672 (ESTABLISHED)
node      3340 onlyoffice   13u  IPv4 6734034      0t0  TCP 127.0.0.1:55992->127.0.0.1:5672 (ESTABLISHED)
node      3340 onlyoffice   14u  IPv4 6727428      0t0  TCP 127.0.0.1:50068->127.0.0.1:5672 (SYN_SENT)
node      3340 onlyoffice   15u  IPv4 6754255      0t0  TCP 127.0.0.1:35738->127.0.0.1:5672 (SYN_SENT)
...
nodeJs has roughly 7k files open:

Code: Select all

$ sudo lsof -n -i -P | grep -c node
7457
Split by ESTABLISHED vs. SYN_SENT it looks like this (in case it is relevant information):

Code: Select all

$ sudo lsof -n -i -P | grep node | grep -c ESTABLISHED
5816
$ sudo lsof -n -i -P | grep node | grep -c SYN_SENT
1223
Any idea where to look further?

We've also setup another onlyoffice running fine. NodeJs has a total of 10 open files there, with 3 connections to RabbitMQ:

Code: Select all

[root@onlyoffice ~]# lsof -i -n -P | grep node
node      24522 onlyoffice   10u  IPv4  80474      0t0  TCP 127.0.0.1:51694->127.0.0.1:6379 (ESTABLISHED)
node      24523 onlyoffice   10u  IPv4  80467      0t0  TCP 127.0.0.1:51692->127.0.0.1:6379 (ESTABLISHED)
node      24523 onlyoffice   12u  IPv6  80550      0t0  TCP *:8000 (LISTEN)
node      24525 onlyoffice   11u  IPv6  80503      0t0  TCP *:8080 (LISTEN)
node      24526 onlyoffice   11u  IPv4  80497      0t0  TCP 127.0.0.1:51698->127.0.0.1:6379 (ESTABLISHED)
node      24574 onlyoffice   11u  IPv4  80532      0t0  TCP 127.0.0.1:51704->127.0.0.1:6379 (ESTABLISHED)
node      24574 onlyoffice   13u  IPv4  80534      0t0  TCP 127.0.0.1:42364->127.0.0.1:5672 (ESTABLISHED)
node      24574 onlyoffice   14u  IPv4  80545      0t0  TCP 127.0.0.1:42366->127.0.0.1:5672 (ESTABLISHED)
node      24575 onlyoffice   12u  IPv4  80513      0t0  TCP 127.0.0.1:42358->127.0.0.1:5672 (ESTABLISHED)
node      24667 onlyoffice   10u  IPv6  81053      0t0  TCP *:3000 (LISTEN)
--------------------------
Our not working System:
OperatingSystem: CentOS 7
OnlyOffice Version: 5.1.4-22

Our working system which we are referencing:
OperatingSystem: CentOS 7
OnlyOffice Version: 5.0.5-26

Carl
Posts: 92
Joined: Thu Apr 12, 2018 10:00 am

Re: Missing (node.js) Upstream when running example project

Post by Carl » Wed Jun 13, 2018 12:10 pm

Hello.

Please try to restart the RabbitMQ service. Also please send us RabbitMQ logs (/var/log/rabbitmq).

Reza.Nazeman
Posts: 5
Joined: Tue Jun 12, 2018 3:59 pm

Re: Missing (node.js) Upstream when running example project

Post by Reza.Nazeman » Wed Jun 13, 2018 1:54 pm

RabbitMQ-Log file:

Code: Select all

closing AMQP connection <0.3519.24> (127.0.0.1:46092 -> 127.0.0.1:5672):
{handshake_timeout,handshake}

=INFO REPORT==== 13-Jun-2018::15:43:48 ===
Stopping RabbitMQ

=INFO REPORT==== 13-Jun-2018::15:43:48 ===
stopped TCP Listener on [::]:5672

=INFO REPORT==== 13-Jun-2018::15:43:51 ===
Halting Erlang VM

=INFO REPORT==== 13-Jun-2018::15:43:53 ===
Starting RabbitMQ 3.3.5 on Erlang R16B03-1
Copyright (C) 2007-2014 GoPivotal, Inc.
Licensed under the MPL.  See http://www.rabbitmq.com/

=INFO REPORT==== 13-Jun-2018::15:43:53 ===
node           : rabbit@iuvm480
home dir       : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash    : ndYDSd0tot75k3beQJ+3rw==
log            : /var/log/rabbitmq/rabbit@iuvm480.log
sasl log       : /var/log/rabbitmq/rabbit@iuvm480-sasl.log
database dir   : /var/lib/rabbitmq/mnesia/rabbit@iuvm480

=INFO REPORT==== 13-Jun-2018::15:43:53 ===
Limiting to approx 924 file handles (829 sockets)

=INFO REPORT==== 13-Jun-2018::15:43:53 ===
Memory limit set to 3128MB of 7822MB total.

=INFO REPORT==== 13-Jun-2018::15:43:53 ===
Disk free limit set to 50MB

=INFO REPORT==== 13-Jun-2018::15:43:53 ===
msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 13-Jun-2018::15:43:53 ===
msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 13-Jun-2018::15:43:53 ===
started TCP Listener on [::]:5672

=INFO REPORT==== 13-Jun-2018::15:43:53 ===
Server startup complete; 0 plugins started.

Reza.Nazeman
Posts: 5
Joined: Tue Jun 12, 2018 3:59 pm

Re: Missing (node.js) Upstream when running example project

Post by Reza.Nazeman » Thu Jun 14, 2018 12:21 pm

This issue was solved by setting umask from 0027 to 0022, and restarting the documentserver services.

Carl
Posts: 92
Joined: Thu Apr 12, 2018 10:00 am

Re: Missing (node.js) Upstream when running example project

Post by Carl » Fri Jun 15, 2018 6:47 am

We are glad the issue has been resolved. Should you have any other questions, let us know.

Post Reply