Try the fastest and simplest way to install ONLYOFFICE

Missing (node.js) Upstream when running example project

Issues during installation and related to database

Missing (node.js) Upstream when running example project

Postby Reza.Nazeman » Tue Jun 12, 2018 4:41 pm

We have an issue running the example project. Clicking on "Create Document" shows only the loading page and end up in the following error after a while:

Loading.PNG
Loading.PNG (95.19 KiB) Viewed 528 times

Loading2.PNG
Loading2.PNG (22.39 KiB) Viewed 528 times


In the nginx logs we found a missing upstream:

Code: Select all
2018/06/12 17:57:16 [error] 1244#0: *9 no live upstreams while connecting to upstream, client: 10.250.52.51, server: , request: "GET /2018-05-22-13-44/doc/-1383711616/c/info?t=1528819037024 HTTP/1.1", upstream: "http://docservice/doc/-1383711616/c/info", host: "onlyoffice.local", referrer: "https://onlyoffice.local/2018-05-22-13-44/web-apps/apps/documenteditor/main/index.html?_dc=2018-05-22-13-44&lang=en&customer=ONLYOFFICE&frameEditorId=iframeEditor"


Checking the file /etc/nginx/includes/onlyoffice-http.conf we found the configured upstream:

Code: Select all
upstream docservice {
  server localhost:8000;
}


After some more research, we figured out that the node-js server should be serving port 8000 with the document-service, but it's not running, port 8000 is missing in the list:

Code: Select all
$ sudo netstat -ntlp | grep node
tcp6       0      0 :::8080                 :::*                    LISTEN      1393/node
tcp6       0      0 :::3000                 :::*                    LISTEN      1805/node


Digging into /var/log/onlyoffice/documentserver/docservice/out.log we find a bunch of these messages:

Code: Select all
[2018-06-12 17:51:07.990] [ERROR] nodeJS - [AMQP] Error: connect EMFILE 127.0.0.1:5672 - Local (undefined:undefined)
    at Object.exports._errnoException (util.js:1020:11)
    at exports._exceptionWithHostPort (util.js:1043:20)
    at connect (net.js:892:16)
    at net.js:1028:7
    at /var/www/onlyoffice/documentserver/server/Common/node_modules/dnscache/lib/index.js:80:28
    at /var/www/onlyoffice/documentserver/server/Common/node_modules/dnscache/lib/cache.js:116:13
    at RawTask.call (/var/www/onlyoffice/documentserver/server/Common/node_modules/asap/asap.js:40:19)
    at flush (/var/www/onlyoffice/documentserver/server/Common/node_modules/asap/raw.js:50:29)
    at _combinedTickCallback (internal/process/next_tick.js:73:7)
    at process._tickCallback (internal/process/next_tick.js:104:9)


The error message indicates, that the docserver is running out of file-descriptors.

The ulimits are pretty high, so checking the open files of nodeJs leads us to here:

Code: Select all
node      1390 onlyoffice   10u  IPv4   20221      0t0  TCP 127.0.0.1:43632->127.0.0.1:6379 (ESTABLISHED)
node      1391 onlyoffice   10u  IPv4   18982      0t0  TCP 127.0.0.1:43630->127.0.0.1:6379 (ESTABLISHED)
node      1393 onlyoffice   11u  IPv6   20206      0t0  TCP *:8080 (LISTEN)
node      1394 onlyoffice   11u  IPv4   21569      0t0  TCP 127.0.0.1:43658->127.0.0.1:6379 (ESTABLISHED)
node      1805 onlyoffice   10u  IPv6   58810      0t0  TCP *:3000 (LISTEN)
node      3340 onlyoffice   11u  IPv4 4305590      0t0  TCP 127.0.0.1:41246->127.0.0.1:6379 (ESTABLISHED)
node      3340 onlyoffice   12u  IPv4 7009241      0t0  TCP 127.0.0.1:54468->127.0.0.1:5672 (ESTABLISHED)
node      3340 onlyoffice   13u  IPv4 6734034      0t0  TCP 127.0.0.1:55992->127.0.0.1:5672 (ESTABLISHED)
node      3340 onlyoffice   14u  IPv4 6727428      0t0  TCP 127.0.0.1:50068->127.0.0.1:5672 (SYN_SENT)
node      3340 onlyoffice   15u  IPv4 6754255      0t0  TCP 127.0.0.1:35738->127.0.0.1:5672 (SYN_SENT)
...


nodeJs has roughly 7k files open:

Code: Select all
$ sudo lsof -n -i -P | grep -c node
7457


Split by ESTABLISHED vs. SYN_SENT it looks like this (in case it is relevant information):

Code: Select all
$ sudo lsof -n -i -P | grep node | grep -c ESTABLISHED
5816
$ sudo lsof -n -i -P | grep node | grep -c SYN_SENT
1223


Any idea where to look further?

We've also setup another onlyoffice running fine. NodeJs has a total of 10 open files there, with 3 connections to RabbitMQ:

Code: Select all
[root@onlyoffice ~]# lsof -i -n -P | grep node
node      24522 onlyoffice   10u  IPv4  80474      0t0  TCP 127.0.0.1:51694->127.0.0.1:6379 (ESTABLISHED)
node      24523 onlyoffice   10u  IPv4  80467      0t0  TCP 127.0.0.1:51692->127.0.0.1:6379 (ESTABLISHED)
node      24523 onlyoffice   12u  IPv6  80550      0t0  TCP *:8000 (LISTEN)
node      24525 onlyoffice   11u  IPv6  80503      0t0  TCP *:8080 (LISTEN)
node      24526 onlyoffice   11u  IPv4  80497      0t0  TCP 127.0.0.1:51698->127.0.0.1:6379 (ESTABLISHED)
node      24574 onlyoffice   11u  IPv4  80532      0t0  TCP 127.0.0.1:51704->127.0.0.1:6379 (ESTABLISHED)
node      24574 onlyoffice   13u  IPv4  80534      0t0  TCP 127.0.0.1:42364->127.0.0.1:5672 (ESTABLISHED)
node      24574 onlyoffice   14u  IPv4  80545      0t0  TCP 127.0.0.1:42366->127.0.0.1:5672 (ESTABLISHED)
node      24575 onlyoffice   12u  IPv4  80513      0t0  TCP 127.0.0.1:42358->127.0.0.1:5672 (ESTABLISHED)
node      24667 onlyoffice   10u  IPv6  81053      0t0  TCP *:3000 (LISTEN)


--------------------------
Our not working System:
OperatingSystem: CentOS 7
OnlyOffice Version: 5.1.4-22

Our working system which we are referencing:
OperatingSystem: CentOS 7
OnlyOffice Version: 5.0.5-26
Reza.Nazeman
 
Posts: 5
Joined: Tue Jun 12, 2018 3:59 pm

Re: Missing (node.js) Upstream when running example project

Postby Carl » Wed Jun 13, 2018 12:10 pm

Hello.

Please try to restart the RabbitMQ service. Also please send us RabbitMQ logs (/var/log/rabbitmq).
Carl
 
Posts: 92
Joined: Thu Apr 12, 2018 10:00 am

Re: Missing (node.js) Upstream when running example project

Postby Reza.Nazeman » Wed Jun 13, 2018 1:54 pm

RabbitMQ-Log file:

Code: Select all
closing AMQP connection <0.3519.24> (127.0.0.1:46092 -> 127.0.0.1:5672):
{handshake_timeout,handshake}

=INFO REPORT==== 13-Jun-2018::15:43:48 ===
Stopping RabbitMQ

=INFO REPORT==== 13-Jun-2018::15:43:48 ===
stopped TCP Listener on [::]:5672

=INFO REPORT==== 13-Jun-2018::15:43:51 ===
Halting Erlang VM

=INFO REPORT==== 13-Jun-2018::15:43:53 ===
Starting RabbitMQ 3.3.5 on Erlang R16B03-1
Copyright (C) 2007-2014 GoPivotal, Inc.
Licensed under the MPL.  See http://www.rabbitmq.com/

=INFO REPORT==== 13-Jun-2018::15:43:53 ===
node           : rabbit@iuvm480
home dir       : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash    : ndYDSd0tot75k3beQJ+3rw==
log            : /var/log/rabbitmq/rabbit@iuvm480.log
sasl log       : /var/log/rabbitmq/rabbit@iuvm480-sasl.log
database dir   : /var/lib/rabbitmq/mnesia/rabbit@iuvm480

=INFO REPORT==== 13-Jun-2018::15:43:53 ===
Limiting to approx 924 file handles (829 sockets)

=INFO REPORT==== 13-Jun-2018::15:43:53 ===
Memory limit set to 3128MB of 7822MB total.

=INFO REPORT==== 13-Jun-2018::15:43:53 ===
Disk free limit set to 50MB

=INFO REPORT==== 13-Jun-2018::15:43:53 ===
msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 13-Jun-2018::15:43:53 ===
msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 13-Jun-2018::15:43:53 ===
started TCP Listener on [::]:5672

=INFO REPORT==== 13-Jun-2018::15:43:53 ===
Server startup complete; 0 plugins started.
Reza.Nazeman
 
Posts: 5
Joined: Tue Jun 12, 2018 3:59 pm

Re: Missing (node.js) Upstream when running example project

Postby Reza.Nazeman » Thu Jun 14, 2018 12:21 pm

This issue was solved by setting umask from 0027 to 0022, and restarting the documentserver services.
Reza.Nazeman
 
Posts: 5
Joined: Tue Jun 12, 2018 3:59 pm

Re: Missing (node.js) Upstream when running example project

Postby Carl » Fri Jun 15, 2018 6:47 am

We are glad the issue has been resolved. Should you have any other questions, let us know.
Carl
 
Posts: 92
Joined: Thu Apr 12, 2018 10:00 am


Return to Installation issues

Who is online

Users browsing this forum: No registered users and 2 guests

cron