CentOS Socket.io 로드밸런싱

공식 문서에는 NginX 설정이 나옴
http://socket.io/docs/using-multiple-nodes/

다중 노드들 사용

#고 load balancing

고정 로드밸런싱

If you plan to distribute the load of connections among different processes or machines, you have to make sure that requests associated with a particular session id connect to the process that originated them.

This is due to certain transports like XHR Polling or JSONP Polling relying on firing several requests during the lifetime of the “socket”.

To illustrate why this is needed, consider the example of emitting an event to all connected clients:

io.emit('hi', 'all sockets');

Chances are that some of those clients might have an active bi-directional communication channel likeWebSocket that we can write to immediately, but some of them might be using long-polling.

If they’re using long polling, they might or might not have sent a request that we can write to. They could be “in between” those requests. In those situations, it means we have to buffer messages in the process. In order for the client to successfully claim those messages when he sends his request, the easiest way is for him to connect to be routed to that same process.

An easy way to do that is by routing clients based on their originating address. An example follows using the NginX server:

#NginX 설정

Within the http { } section of your nginx.conf file, you can declare a upstream section with a list of Socket.IO process you want to balance load between:

upstream io_nodes {
  ip_hash;
  server 127.0.0.1:6001;
  server 127.0.0.1:6002;
  server 127.0.0.1:6003;
  server 127.0.0.1:6004;
}

Notice the ip_hash instruction that indicates the connections will be sticky.

In the same http { } section, you can declare a server { } that points to this upstream. In order for NginX to support and forward the WebSocket protocol, we explicitly pass along the required Upgrade headers:

server {
  listen 3000;
  server_name io.yourhost.com;
  location / {
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $host;
    proxy_http_version 1.1;
    proxy_pass http://io_nodes;
  }
}

Make sure you also configure worker_processes in the topmost level to indicate how many workers NginX should use. You might also want to look into tweaking the worker_connections setting within the events { }block.

#Node.JS 클러스터 사용

Just like NginX, Node.JS comes with built-in clustering support through the cluster module.

Fedor Indutny has created a module called sticky session that ensures file descriptors (ie: connections) are routed based on the originating remoteAddress (ie: IP).

#노드간 이벤트 전달

Now that you have multiple Socket.IO nodes accepting connections, if you want to broadcast events to everyone (or even everyone in a certain room) you’ll need some way of passing messages between processes or computers.

The interface in charge of routing messages is what we call the Adapter. You can implement your own on top of the socket.io-adapter (by inheriting from it) or you can use the one we provide on top of Redis: socket.io-redis:

var io = require('socket.io')(3000);
var redis = require('socket.io-redis');
io.adapter(redis({ host: 'localhost', port: 6379 }));

If you want to pass messages to it from non-socket.io processes, you should look into “Sending messages from the outside-world”.

소켓접속이 안되는 이슈가 발생하여

로그밸런서 서버 로그를 살펴봄

tail -20 /var/log/nginx/error.log
젠장 너무 많고, 필터링이 필요...

여러가지 삽질 시도하다가

CentOS 에는 권한 관련 골치아픈 존재인 SELinux 를 뒤늦게 상기하고

$ setenforce 0

해보니 소켓접속 아주 잘됨 = Reverse Proxy 잘 동작

보안상 놔두는것이 좋으니 다시 셋팅

$ setenforce 1

과연 SELinux로 막힌 소켓은 어떻게 권한을 줘야 하나
구글신에게 문의, 조회하다 훌륭한 글 발견 !!
http://axilleas.me/en/blog/2013/selinux-policy-for-nginx-and-gitlab-unix-socket-in-fedora-19/

오류 사냥

오류 발생시 처음 할 일은 로그를 보는것.

Nginx

/var/log/nginx/gitlab_error.log 에서 아래 오류 반복을 찾을 수 있었음:

2013/08/26 21:43:01 [crit] 2597#0: *50 connect() to unix:/home/git/gitlab/tmp/sockets/gitlab.socket failed (13: Permission denied) while connecting to upstream, client 12.34.56.78, server: fedora.axilleas.me, request: "GET /users/sign_in HTTP/1.1", upstream: "http://unix:/home/git/gitlab/tmp/sockets/gitlab.socket:/users/sign_in", host: "fedora.axilleas.me"

So we got a permission denied while nginx is trying to connect to the unix socket of GitLab. After some hours searching and reading answers in stackoverflow, it sroke to me to check whether SELinux is to blame. I set it to permissive mode with setenforce 0 and voila, nginx 가 갑자기 요청을 받았을 때.

SELinux 너 이 악당 blocker

I remembered the awesome introductory guide of SELinux at CentOS wiki, which I had used when rewriting the CentOS installation guide for GitLab and immediately started reading.

기본적으로, SELinux 로그메세지는 /var/log/audit/audit.log 파일에 쓰여지며 리눅스 Auditing 시스템인 auditd가 행한다. If the auditd daemon is not running, then messages are written to /var/log/messages. SELinux log messages are labeled with the AVC keyword so that they might be easily filtered from other messages, as with grep.

So, by greping nginx in /var/log/audit/audit.log I found those relative AVC messages, which indicate indeed a denial of nginx connection to gitlab.socket.

type=AVC msg=audit(1377542938.307:248364): avc:  denied  { write } for  pid=2597 comm="nginx" name="gitlab.socket" dev="vda1" ino=1180273 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_sys_content_t:s0 tclass=sock_file
type=AVC msg=audit(1377542938.307:248364): avc:  denied  { connectto } for  pid=2597 comm="nginx" path="/home/git/gitlab/tmp/sockets/gitlab.socket" scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:initrc_t:s0 tclass=unix_stream_socket

AVC 메세지들을 빼고 볼 수 있게 해주는 audit2allow 라고 불리는 도구를 사용하자.
설치한적이 없다면 yum install -y policycoreutils-devel 패키지 설치하여 탑재하자.

grep nginx /var/log/audit/audit.log | audit2allow

그리고 결과는:

#============= httpd_t ==============

#!!!! This avc is allowed in the current policy
allow httpd_t http_cache_port_t:tcp_socket name_connect;

#!!!! This avc is allowed in the current policy
allow httpd_t httpd_log_t:file setattr;

#!!!! This avc is allowed in the current policy
allow httpd_t httpd_sys_content_t:sock_file write;

#!!!! This avc is allowed in the current policy
allow httpd_t initrc_t:unix_stream_socket connectto;

#!!!! This avc is allowed in the current policy
allow httpd_t user_home_dir_t:dir search;

#!!!! This avc is allowed in the current policy
allow httpd_t user_home_t:dir { search getattr };

#!!!! This avc is allowed in the current policy
allow httpd_t user_home_t:sock_file write;

#!!!! This avc is allowed in the current policy
allow httpd_t var_run_t:file { read write };

These are the policies that should be used with SELinux. Notice that user_home is essential since GitLab's APP_ROOT is in /home/git/. Similarly, you notice a policy related to the denied socket connection: unix_stream_socket connectto.

커스텀 SELinux 정책 모듈 생성

After all the investigation we are closer to the solution. All we have to do is use audit2allow to generate a set of policy rules that would allow the required actions. We can generate a local nginx Type Enforcement policy file (nginx.te):

grep nginx /var/log/audit/audit.log | audit2allow -m nginx > nginx.te
cat nginx.te


module nginx 1.0;

require {
    type var_run_t;
    type user_home_dir_t;
    type httpd_log_t;
    type httpd_t;
    type user_home_t;
    type httpd_sys_content_t;
    type initrc_t;
    type http_cache_port_t;
    class sock_file write;
    class unix_stream_socket connectto;
    class dir { search getattr };
    class file { read write setattr };
    class tcp_socket name_connect;
}

#============= httpd_t ==============

#!!!! This avc is allowed in the current policy
allow httpd_t http_cache_port_t:tcp_socket name_connect;
allow httpd_t httpd_log_t:file setattr;
allow httpd_t httpd_sys_content_t:sock_file write;
allow httpd_t initrc_t:unix_stream_socket connectto;

#!!!! This avc is allowed in the current policy
allow httpd_t user_home_dir_t:dir search;

#!!!! This avc is allowed in the current policy
allow httpd_t user_home_t:dir { search getattr };
allow httpd_t user_home_t:sock_file write;
allow httpd_t var_run_t:file { read write };

아직 끝나지 않았다, as this is a file for review only. We can then go ahead and use audit2allow to make a custom policy module to allow these actions:

grep nginx /var/log/audit/audit.log | audit2allow -M nginx
semodule -i nginx.pp

올바로 로딩된 정책 모듈 검사는 semodule -l. 명령으로 로딩된 모듈들을 리스팅하여 할 수 있다.

이후, setenforce 1.명령으로 SELinux 활성화를 기억하라.

TL;DR (요약)

모든 nginx 502 문제를 고치려면 root 로 아래 명령들 실행:

$ yum install -y policycoreutils-{python,devel}
$ grep nginx /var/log/audit/audit.log | audit2allow -M nginx
$ semodule -i nginx.pp

journald 에 SELinux 오류 메세지 통합

In a very interesting article, Dan Walsh explains how this whole process of error hunting will be much easier with Fedora 20. I urge you to read it.

With the upcoming changes, the error would have appeared at systemd's status log:

systemctl status nginx

그리고 가능한 해결책은:

journalctl  -r -o verbose -u nginx.service

HAProxy 설정은
http://www.server-world.info/en/note?os=CentOS_7&p=haproxy

위 SELinux 레퍼런스 활용:

$ yum install -y policycoreutils-{python,devel} $ grep haproxy /var/log/audit/audit.log | audit2allow -M haproxy $ semodule -i haproxy.pp