Installing Nodejs on CentOS/Redhat Linux

Follow these steps to install nodejs on Centos. 
yum install -y gcc-c++ make

  sudo yum install -y gcc-c++ make

  curl -sL https://rpm.nodesource.com/setup_6.x | sudo -E bash -

  yum install -y gcc-c++ make

  sudo yum install -y gcc-c++ make

  yum install nodejs

  sudo yum install nodejs

  node -v
    v6.12.0
[centos@ip-174-35-28-152 ~]$  npm -v
3.10.10

Read More

Amazon Virtual Private Network (VPC) in short

Amazon Virtual Network Cloud (VPC) enables enables you to build a secure virtual network in the amazon cloud.

For creating VPC, no hardware or physical data centres required. You can define your own network space and control how your network and EC2 resources inside your network are exposed to the internet.

What sort of security options do i have over my virtual private network ?

You can leverage the security options in AWS VPC to provide more granular access both to and from the EC2 instances in your virtual network.

You can create a public facing subnet for your web servers that have access to internet , private facing subnet with no internet access for backend systems such as databases, app servers  etc.

What if i don’t want to connect my secure network to the public facing internet to restrict within corporate network ?

You can create a hardware virtual private network connection between your corporate data centre and  your Amazon VPC.

It can be used to enable the inbound and outbound filtering at the instance level and subnet level.

For storing  data , you can use S3 and restrict access ,so that it is only accessible from instances in your VPC.

Read More

Error executing 'up': Cloudformation failure waiting for 'CREATE_COMPLETE'. State is 'ROLLBACK_COMPLETE'

ecs-cli up --keypair mycredentials --capability-iam --size 5 --azs us-east-1a,us-east-1b --instance-type m4.large

got this issue.

Failure event  reason="Parameter validation failed: parameter value mycredentials for parameter name KeyName does not exist. Rollback requested by user." resourceType="AWS::CloudFormation::Stack"

ERRO[0045] Error executing 'up': Cloudformation failure waiting for 'CREATE_COMPLETE'. State is 'ROLLBACK_COMPLETE' 

Read More

S3 bucket - repository/tag: invalid reference format.

I am unable to backup docker volume to S3. Trying to run the docker command.

 docker run -d --name 945e1345eed616bbd2a5dd920df2d -e BACKUP_INTERVAL=1m \ S3_BUCKET=33222a9e09 -e AWS_ACCESS_KEY_ID=DFHFHGJHS2KIADCGGD7IBVZQ -e AWS_SECRET_ACCESS_KEY=mbfdbd6kvUgpPlH/7H5/jfJRJnaX1RTJBDF



docker: Error parsing reference: " S3_BUCKET=33222a9e09" is not a valid repository/tag: invalid reference format.

docker: Error parsing reference: "S3_BUCKET=hfghdgfhjdf" is not a valid repository/tag: invalid reference format.

How to configure s3 bucket to the docker  volume backup

Read More

Cloud Watch Alarms- policy issue

Your Auto Scaling policy ScaleOutPolicy could not be found at this time. Please try reselecting this alarm. If the policy does not exist you may modify this alarm to set up a valid action.

Read More

AWS S3 Tutorial

Amazon Simple Storage Service is also known as Amazon S3. It is highly scalable object storage in the cloud securely.
It is used to store any amount of data.
MultiPart form uploading to AWS

S3 is Object Storage with a simple web service interface to store and retrive any amount of data from anywhere on the web.

What is multi part upload ?
1. Initiate the Multi part Upload
2. Seperate the object into multiple parts.
3. Upload the parts in any order, one at a time or in the parallel.

Date transmission - Browser going to App server and going to S3.
https://s3.amazonaws.com/upload-demo-images/5e64truyt47653745tr46tr736t4rt4r3t46

Nodejs code :
Install the dependencies.
npm install s3-stream-upload


var UploadStream = require("s3-stream-upload");
var S3 = require("aws-sdk").S3;

var key = "file.mp3";
var s3 = new S3();

fs.readFileStream(__dirname + "/file.mp3")
  .pipe(UploadStream(s3, { Bucket: "my-S3-bucket", Key: key }))
  .on("error", function (err) {
    console.error(err);
  })
  .on("finish", function () {
    console.log("File uploaded!");
  });

my sample s3 url - 'https://s3.amazonaws.com/upload-demo-images/';

Read More

Aws Security Groups

The very first thing to ensure that the unwanted or unauthorized traffic doesn't get in into the instances using Security Group.

What is a Security Group?
It is a AWS firewall solution which filters the incoming and outgoing traffic from an instance.
Filtering is done based upon the ip protocols,ports and source/destination ip addresses.

At the backend requires X.509 certificate and key to authorise changes.

How to configure security groups while creating instance.
You can configure security groups while creating instance or anytime after creating instance.

Figure 1.0

Figure 1.1

Here you can configure security group.

Start configure security rules. Security rules for set of firewall rules that control the traffic. what kind of traffic can reach the instance.

The rules we are going to configure are Inbound rules.

SSH/RDP:

If we use linux based machine, configure SSH. SSH stands for secure shell. Secure access to the remote machine.It always communicates over port 22.It is static and you cannot change the port.

If it is Windows based machine, configure RDP. By default it will give root access to the machine.

If we select source anywhere, across internet can access the machine via SSH. This is quite dangerous. 

You can choose custom ip for security purpose. 

Web Traffic:

Web traffic can come from any source.

0.0.0.0/0     -> allows all the traffic.which is not secure.

HTTP communicates over port 80.

HTTPS always communicates over port 443. 

Depends on what traffic you want to allow in, you can add rule here.

Figure 1.2

Figure 1.3

Read More

aws instance types

General Purpose (T2, M3, M4 class):

These provides overall balance of Ram, CPU and provide burstable performance.
IOP starts from 100 and upto 3000.

Compute Optimized (C3 and C4 class):

It is meant for hude traffic.

Storage Optimized(I3  and D2 class):

It is for large datawarehouse.

Memory Optimized(X1, R3 and R4 class):

It is for Memcached and enterprise apps.

Accelerated Computing(P2, G3 and F1 class):

It is for general-purpose GPU compute applications,graphics-intensive applications and FPGAs.

Read More

Unabme to run docker without sudo

I am unable to run docker without sudo.

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.25/containers/json: dial unix /var/run/docker.sock: connect: permission denied


I added docker to user group

sudo usermod -aG docker $USER

Still i am unable to run without sudo.

Read More

docker Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Warning: failed to get default registry endpoint from daemon (Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?). Using system default: https://index.docker.io/v1/

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Read More

Jupyterhub is not running on ubuntu 16.04

$ jupyterhub

/home/ubuntu/jupyterhub_config.py:6: UserWarning: IPython.utils.localinterfaces has moved to jupyter_client.localinterfaces

  from IPython.utils.localinterfaces import public_ips

[I 2017-08-07 18:36:18.545 JupyterHub app:724] Loading cookie_secret from /home/ubuntu/jupyterhub_cookie_secret

[W 2017-08-07 18:36:18.576 JupyterHub app:365] 

    Generating CONFIGPROXY_AUTH_TOKEN. Restarting the Hub will require restarting the proxy.

    Set CONFIGPROXY_AUTH_TOKEN env or JupyterHub.proxy_auth_token config to avoid this message.

    

[W 2017-08-07 18:36:18.579 JupyterHub app:864] No admin users, admin interface will be unavailable.

[W 2017-08-07 18:36:18.580 JupyterHub app:865] Add any administrative users to `c.Authenticator.admin_users` in config.

[I 2017-08-07 18:36:18.580 JupyterHub app:892] Not using whitelist. Any authenticated user will be allowed.

[I 2017-08-07 18:36:18.593 JupyterHub app:1453] Hub API listening on http://172.31.28.236:8081/hub/

[W 2017-08-07 18:36:18.596 JupyterHub app:1174] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...

[I 2017-08-07 18:36:18.596 JupyterHub app:1176] Starting proxy @ http://*:8000/

/usr/bin/env: ‘node’: No such file or directory

[C 2017-08-07 18:36:19.605 JupyterHub app:1459] Failed to start proxy

    Traceback (most recent call last):

      File "/home/ubuntu/miniconda3/lib/python3.6/site-packages/jupyterhub/app.py", line 1457, in start

        yield self.start_proxy()

      File "/home/ubuntu/miniconda3/lib/python3.6/site-packages/jupyterhub/app.py", line 1197, in start_proxy

        _check()

      File "/home/ubuntu/miniconda3/lib/python3.6/site-packages/jupyterhub/app.py", line 1193, in _check

        raise e

    RuntimeError: Proxy failed to start with exit code 127

solution:

sudo apt-get install npm nodejs-legacy

Read More

Install NodeJS on Ubuntu 16.04

1.  sudo apt-get update

2.  sudo apt-get install nodejs

3.  sudo apt-get install npm

 sudo apt-get install nodejs

Reading package lists... Done

Building dependency tree       

Reading state information... Done

The following additional packages will be installed:

  libuv1

The following NEW packages will be installed:

  libuv1 nodejs

0 upgraded, 2 newly installed, 0 to remove and 23 not upgraded.

Need to get 3218 kB of archives.

After this operation, 13.4 MB of additional disk space will be used.

Do you want to continue? [Y/n] Y

Get:1 http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu xenial/universe amd64 libuv1 amd64 1.8.0-1 [57.4 kB]

Get:2 http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 nodejs amd64 4.2.6~dfsg-1ubuntu4.1 [3161 kB]

Fetched 3218 kB in 0s (61.6 MB/s)

perl: warning: Setting locale failed.

perl: warning: Please check that your locale settings:

LANGUAGE = (unset),

LC_ALL = (unset),

LC_CTYPE = "UTF-8",

LANG = "en_US.UTF-8"

    are supported and installed on your system.

perl: warning: Falling back to a fallback locale ("en_US.UTF-8").

locale: Cannot set LC_CTYPE to default locale: No such file or directory

locale: Cannot set LC_ALL to default locale: No such file or directory

Selecting previously unselected package libuv1:amd64.

(Reading database ... 51035 files and directories currently installed.)

Preparing to unpack .../libuv1_1.8.0-1_amd64.deb ...

Unpacking libuv1:amd64 (1.8.0-1) ...

Selecting previously unselected package nodejs.

Preparing to unpack .../nodejs_4.2.6~dfsg-1ubuntu4.1_amd64.deb ...

Unpacking nodejs (4.2.6~dfsg-1ubuntu4.1) ...

Processing triggers for libc-bin (2.23-0ubuntu9) ...

Processing triggers for man-db (2.7.5-1) ...

Setting up libuv1:amd64 (1.8.0-1) ...

Setting up nodejs (4.2.6~dfsg-1ubuntu4.1) ...

update-alternatives: using /usr/bin/nodejs to provide /usr/bin/js (js) in auto mode

Processing triggers for libc-bin (2.23-0ubuntu9) ...

ubuntu@ip-192-31-28-436:~$ sudo apt-get install npm

Reading package lists... Done

Building dependency tree       

Reading state information... Done

The following additional packages will be installed:

  binutils build-essential cpp cpp-5 dpkg-dev fakeroot g++ g++-5 gcc gcc-5 gyp

  javascript-common libalgorithm-diff-perl libalgorithm-diff-xs-perl

  libalgorithm-merge-perl libasan2 libatomic1 libc-dev-bin libc6-dev libcc1-0

  libcilkrts5 libdpkg-perl libfakeroot libfile-fcntllock-perl libgcc-5-dev

  libgomp1 libisl15 libitm1 libjs-inherits libjs-jquery libjs-node-uuid

  libjs-underscore liblsan0 libmpc3 libmpx0 libpython-stdlib

  libpython2.7-minimal libpython2.7-stdlib libquadmath0 libssl-dev libssl-doc

  libstdc++-5-dev libtsan0 libubsan0 libuv1-dev linux-libc-dev make

  manpages-dev node-abbrev node-ansi node-ansi-color-table node-archy

  node-async node-block-stream node-combined-stream node-cookie-jar

  node-delayed-stream node-forever-agent node-form-data node-fstream

  node-fstream-ignore node-github-url-from-git node-glob node-graceful-fs

  node-gyp node-inherits node-ini node-json-stringify-safe node-lockfile

  node-lru-cache node-mime node-minimatch node-mkdirp node-mute-stream

  node-node-uuid node-nopt node-normalize-package-data node-npmlog node-once

  node-osenv node-qs node-read node-read-package-json node-request node-retry

  node-rimraf node-semver node-sha node-sigmund node-slide node-tar

  node-tunnel-agent node-underscore node-which nodejs-dev python

  python-minimal python-pkg-resources python2.7 python2.7-minimal zlib1g-dev

Suggested packages:

  binutils-doc cpp-doc gcc-5-locales debian-keyring g++-multilib

  g++-5-multilib gcc-5-doc libstdc++6-5-dbg gcc-multilib autoconf automake

  libtool flex bison gdb gcc-doc gcc-5-multilib libgcc1-dbg libgomp1-dbg

  libitm1-dbg libatomic1-dbg libasan2-dbg liblsan0-dbg libtsan0-dbg

  libubsan0-dbg libcilkrts5-dbg libmpx0-dbg libquadmath0-dbg apache2

  | lighttpd | httpd glibc-doc libstdc++-5-doc make-doc node-hawk

  node-aws-sign node-oauth-sign node-http-signature debhelper python-doc

  python-tk python-setuptools python2.7-doc binfmt-support

The following NEW packages will be installed:

  binutils build-essential cpp cpp-5 dpkg-dev fakeroot g++ g++-5 gcc gcc-5 gyp

  javascript-common libalgorithm-diff-perl libalgorithm-diff-xs-perl

  node-gyp node-inherits node-ini node-json-stringify-safe node-lockfile

  node-lru-cache node-mime node-minimatch node-mkdirp node-mute-stream

  node-node-uuid node-nopt node-normalize-package-data node-npmlog node-once

  node-osenv node-qs node-read node-read-package-json node-request node-retry

  node-rimraf node-semver node-sha node-sigmund node-slide node-tar

  node-tunnel-agent node-underscore node-which nodejs-dev npm python

  python-minimal python-pkg-resources python2.7 python2.7-minimal zlib1g-dev

0 upgraded, 102 newly installed, 0 to remove and 23 not upgraded.

Need to get 47.7 MB of archives.

After this operation, 188 MB of additional disk space will be used.

Do you want to continue? [Y/n] Y

Get:1 http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 libpython2.7-minimal amd64 2.7.12-1ubuntu0~16.04.1 [339 kB]

Get:2 http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 python2.7-minimal 

Unpacking node-npmlog (0.0.4-1) ...

Selecting previously unselected package node-osenv.

Preparing to unpack .../node-osenv_0.1.0-1_all.deb ...

Unpacking node-osenv (0.1.0-1) ...

Selecting previously unselected package node-tunnel-agent.

Preparing to unpack .../node-tunnel-agent_0.3.1-1_all.deb ...

Unpacking node-tunnel-agent (0.3.1-1) ...

Selecting previously unselected package node-json-stringify-safe.

Preparing to unpack .../node-json-stringify-safe_5.0.0-1_all.deb ...

Unpacking node-json-stringify-safe (5.0.0-1) ...

Selecting previously unselected package node-qs.

Preparing to unpack .../node-qs_2.2.4-1_all.deb ...

Unpacking node-qs (2.2.4-1) ...

Selecting previously unselected package node-request.

Preparing to unpack .../node-request_2.26.1-1_all.deb ...

Unpacking node-request (2.26.1-1) ...

Selecting previously unselected package node-semver.

Preparing to unpack .../node-semver_2.1.0-2_all.deb ...

Unpacking node-semver (2.1.0-2) ...

Selecting previously unselected package node-tar.

Preparing to unpack .../node-tar_1.0.3-2_all.deb ...

Unpacking node-tar (1.0.3-2) ...

Selecting previously unselected package node-which.

Preparing to unpack .../node-which_1.0.5-2_all.deb ...

Unpacking node-which (1.0.5-2) ...

Selecting previously unselected package node-gyp.

Preparing to unpack .../node-gyp_3.0.3-2ubuntu1_all.deb ...

Unpacking node-gyp (3.0.3-2ubuntu1) ...

Selecting previously unselected package node-ini.

Preparing to unpack .../node-ini_1.1.0-1_all.deb ...

Unpacking node-ini (1.1.0-1) ...

Selecting previously unselected package node-lockfile.

Preparing to unpack .../node-lockfile_0.4.1-1_all.deb ...

Unpacking node-lockfile (0.4.1-1) ...

Selecting previously unselected package node-mute-stream.

Preparing to unpack .../node-mute-stream_0.0.4-1_all.deb ...

Unpacking node-mute-stream (0.0.4-1) ...

Selecting previously unselected package node-normalize-package-data.

Preparing to unpack .../node-normalize-package-data_0.2.2-1_all.deb ...

Unpacking node-normalize-package-data (0.2.2-1) ...

Selecting previously unselected package node-read.

Preparing to unpack .../node-read_1.0.5-1_all.deb ...

Unpacking node-read (1.0.5-1) ...

Selecting previously unselected package node-read-package-json.

Preparing to unpack .../node-read-package-json_1.2.4-1_all.deb ...

Unpacking node-read-package-json (1.2.4-1) ...

Selecting previously unselected package node-retry.

Preparing to unpack .../node-retry_0.6.0-1_all.deb ...

Unpacking node-retry (0.6.0-1) ...

Setting up node-lru-cache (2.3.1-1) ...

Setting up node-sigmund (1.0.0-1) ...

Setting up node-minimatch (1.0.0-1) ...

Setting up node-fstream-ignore (0.0.6-2) ...

Setting up node-github-url-from-git (1.1.1-1) ...

Setting up node-once (1.1.1-1) ...

Setting up node-glob (4.0.5-1) ...

Setting up nodejs-dev (4.2.6~dfsg-1ubuntu4.1) ...

Setting up node-nopt (3.0.1-1) ...

Setting up node-npmlog (0.0.4-1) ...

Setting up node-osenv (0.1.0-1) ...

Setting up node-tunnel-agent (0.3.1-1) ...

Setting up node-json-stringify-safe (5.0.0-1) ...

Setting up node-qs (2.2.4-1) ...

Setting up node-request (2.26.1-1) ...

Setting up node-semver (2.1.0-2) ...

Setting up node-tar (1.0.3-2) ...

Setting up node-which (1.0.5-2) ...

Setting up node-gyp (3.0.3-2ubuntu1) ...

Setting up node-ini (1.1.0-1) ...

Setting up node-lockfile (0.4.1-1) ...

Setting up node-mute-stream (0.0.4-1) ...

Setting up node-normalize-package-data (0.2.2-1) ...

Setting up node-read (1.0.5-1) ...

Setting up node-read-package-json (1.2.4-1) ...

Setting up node-retry (0.6.0-1) ...

Setting up node-sha (1.2.3-1) ...

Setting up node-slide (1.1.4-1) ...

Setting up npm (3.5.2-0ubuntu4) ...

Processing triggers for libc-bin (2.23-0ubuntu9) ...

Read More

no space left on device docker

failed to register layer: Error processing tar file(exit status 1): write /opt/conda/lib/python3.5/site-packages/sympy/core/tests/test_args.py: no space left on device

Read More

install docker-engine issue

sudo apt install docker-engine

Reading package lists... Done

Building dependency tree       

Reading state information... Done

Some packages could not be installed. This may mean that you have

requested an impossible situation or if you are using the unstable

distribution that some required packages have not yet been created

or been moved out of Incoming.

The following information may help to resolve the situation:

The following packages have unmet dependencies:

 docker-engine : Depends: libsystemd-journal0 (>= 201) but it is not installable

                 Recommends: aufs-tools but it is not going to be installed

E: Unable to correct problems, you have held broken packages.

Read More

The program 'docker' is currently not installed.

docker -v

The program 'docker' is currently not installed. You can install it by typing:

sudo apt install docker.io

(my_env) ubuntu@ip-192-31-19-128:/tmp$ sudo apt install docker.io

Reading package lists... Done

Building dependency tree       

Reading state information... Done

The following additional packages will be installed:

  bridge-utils cgroupfs-mount containerd runc ubuntu-fan

Suggested packages:

  mountall aufs-tools debootstrap docker-doc rinse zfs-fuse | zfsutils

The following NEW packages will be installed:

  bridge-utils cgroupfs-mount containerd docker.io runc ubuntu-fan

0 upgraded, 6 newly installed, 0 to remove and 22 not upgraded.

Need to get 16.4 MB of archives.

After this operation, 83.6 MB of additional disk space will be used.

Do you want to continue? [Y/n] Y

Get:1 http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu xenial/main amd64 bridge-utils amd64 1.5-9ubuntu1 [28.6 kB]

Get:2 http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu xenial/universe amd64 cgroupfs-mount all 1.2 [4970 B]

Get:3 http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 runc amd64 1.0.0~rc2+docker1.12.6-0ubuntu1~16.04.1 [1479 kB]

Get:4 http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 containerd amd64 0.2.5-0ubuntu1~16.04.1 [4041 kB]

Get:5 http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 docker.io amd64 1.12.6-0ubuntu1~16.04.1 [10.8 MB]

Get:6 http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 ubuntu-fan all 0.9.2 [30.7 kB]

Fetched 16.4 MB in 1s (12.2 MB/s)

perl: warning: Setting locale failed.

perl: warning: Please check that your locale settings:

LANGUAGE = (unset),

LC_ALL = (unset),

LC_CTYPE = "UTF-8",

LANG = "en_US.UTF-8"

    are supported and installed on your system.

perl: warning: Falling back to a fallback locale ("en_US.UTF-8").

locale: Cannot set LC_CTYPE to default locale: No such file or directory

locale: Cannot set LC_ALL to default locale: No such file or directory

Selecting previously unselected package bridge-utils.

(Reading database ... 62405 files and directories currently installed.)

Preparing to unpack .../bridge-utils_1.5-9ubuntu1_amd64.deb ...

Unpacking bridge-utils (1.5-9ubuntu1) ...

Selecting previously unselected package cgroupfs-mount.

Preparing to unpack .../cgroupfs-mount_1.2_all.deb ...

Unpacking cgroupfs-mount (1.2) ...

Selecting previously unselected package runc.

Preparing to unpack .../runc_1.0.0~rc2+docker1.12.6-0ubuntu1~16.04.1_amd64.deb ...

Unpacking runc (1.0.0~rc2+docker1.12.6-0ubuntu1~16.04.1) ...

Selecting previously unselected package containerd.

Preparing to unpack .../containerd_0.2.5-0ubuntu1~16.04.1_amd64.deb ...

Unpacking containerd (0.2.5-0ubuntu1~16.04.1) ...

Selecting previously unselected package docker.io.

Preparing to unpack .../docker.io_1.12.6-0ubuntu1~16.04.1_amd64.deb ...

Unpacking docker.io (1.12.6-0ubuntu1~16.04.1) ...

Selecting previously unselected package ubuntu-fan.

Preparing to unpack .../ubuntu-fan_0.9.2_all.deb ...

Unpacking ubuntu-fan (0.9.2) ...

Processing triggers for man-db (2.7.5-1) ...

Processing triggers for ureadahead (0.100.0-19) ...

Processing triggers for systemd (229-4ubuntu19) ...

Setting up bridge-utils (1.5-9ubuntu1) ...

locale: Cannot set LC_CTYPE to default locale: No such file or directory

locale: Cannot set LC_ALL to default locale: No such file or directory

Setting up cgroupfs-mount (1.2) ...

Setting up runc (1.0.0~rc2+docker1.12.6-0ubuntu1~16.04.1) ...

Setting up containerd (0.2.5-0ubuntu1~16.04.1) ...

Setting up docker.io (1.12.6-0ubuntu1~16.04.1) ...

Adding group `docker' (GID 116) ...

Done.

Setting up ubuntu-fan (0.9.2) ...

Processing triggers for systemd (229-4ubuntu19) ...

Processing triggers for ureadahead (0.100.0-19) ...

Read More

conda install traitlets tornado jinja2 sqlalchemy 

conda: command not found

Read More

Permission error- Unable to ssh into Amazon EC2 instance

ssh -i ec2-instance-cert.pem root@publicip or public dns

@@@@@@@@@@@@@@@@@@@@@@@@######@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ ###        WARNING: UNPROTECTED PRIVATE KEY FILE! ###   @
@@@@@@@@@@@@@@@@@@@@@@@@######@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Permissions 0644 for 'test.pem' are too open.

It is required that your private key files are NOT accessible by others.

Solution :
key must not be publicly viewable for SSH to work. Use this command if required

$ chmod 400 ec2-instance-cert.pem

Read More

Mongoose

It is an object modeling tool for nodejs that connects MongoDB.Mongoose take the models and maps them to MongoDB database.

install required dependencies
npm install mongoose

var mongoose = require('mongoose');



/*without callback */
mongoose.connect('localhost','mydatabase');

/* with callback */ create a connection object and bind two callabcks.

var db = mongoose.connection;
db.on('open',function callback(){
//connection success
});
db.on('error',function(msg){
//connection error
});

Schema Creation :

var empSchema = mongoose.Schema({
name: String,
designation : String,
location: String,
Skills: String
});

Model Creation :

var Employee =  mongoose.model('employee',empSchema);

Saving the model :

employee.save(function(err,emp){
if(err){
//error handling
}else{
//success
}
});

Reading the model :

var Employee =  mongoose.model('employee',empSchema);

Employee.find(function(err,emp){
if(err){
//error handling
}else{
//success
}
});

Reading specific model :


Employee.find({name:'SrinivasNidadavolu'},function(err,emp){
if(err){
//error handling
}else{
//success
}
});

output:

[{name:'SrinivasNidadavolu',designation:'software engineer',location:'building number 14,Raheja Mindpsace, Hyderabad',Skills:'Java,C++,AWS,AZURE,Containers,cloud specialist'}]

Read More

Elastic Load Balancing

Elastic Load Balancing automatically distributes the incoming application traffic across multiple was ec2 instances.

Two types of load balancers :
Application Load Balancer
Classic Load Balancer

Elastic Load Balancing :
It is in-region load balancing service and distributed traffic  across multiple availability zones -HTTP/S, TCP/S.
Fully fault tolerant
Buit in health check

Application Load Balancer Features:
• It has Containerized Application Support
• It has Content-Based Routing
• It support for  HTTP/2
• Contains Delete Protection
• Supports WebSockets
• Supports Layer-7 Load Balancing
• Supports Web Application Firewall (WAF)
• Contains Request Tracing

Classic Load Balancer Features:
• Contains Health Checks
• It has High Availability
• Supports Security Features
• Supports Sticky Sessions
• Supports SSL Offloading
• Support for Layer 4 or 7 Load Balancing
• It has Operational Monitoring
• provides Logging
• provides IPv6 Support

Read More

Creating Database MongoDB

MongoDB is document based database.
Creating Database NodeJS - MongoDB

Once mongoDB is up and running,

var mongoDB =  require('mongodb');


var mydbserver =  new mongodb.server('localhost', 27017, {auto_reconnect:true});

/* new mongodb.Db() creates a new database */
/* w:1 sets the write concern */

var database =  new mongodb.Db('eCommerceDb',mydbserver, {w:1});



Collections :

Relational databases uses tables to store data. MongoDB stores data as document format uses collection. It is NoSql type database.

Collections are represented as JSON format.

Read More

Installing MongoDB

Mac Installation :

The easy way to install MongoDB is by using package manager like Homebrew  or MacPorts.

$ sudo brew install mongodb

or 

$ sudo port install mongodb

Setup the data directory

$ sudo mkdir -p /data/db

setup appropriate permissions

$ sudo chown 'id -u' /data/db

Ubuntu Installation :

To authenticate apt-get package, install the public GPG key from MongoDB creators.

$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10

create a file /etc/apt/sources.list.d/10gen.list  and  add the below text

deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen

Install latest stable build of MongoDB

$ sudo apt-get update

$ sudo apt-get install mongodb-10gen

Windows Installation :

Download the latest stable build from https://www.mongodb.com/download-center#community

Make sure appropriate version 64 bit or 32 bit

Install with the installation file.

create the data directory 

mkdir data

mkdir data\db

Running MongoDB

On Mac :

mongod

On Ubuntu :

$ sudo service mongodb start

On Windows :

c:\mongodb\bin\mongod.exe

Read More

Features of MongoDB

One of the  main reasons  to use NOSQL database is built in sharing capabilities.

Sharing allows to distribute single database across cluster of machines.

It means, distribute the load rather than purchasing the large server. 

Sharing also help in server handling big data. The data stored in database is increasing exponentially. 

Some databases are very large for each server. So distribution across cluster is necessary. 

No sql databases are lot easier to maintain with the administration tools.

It also supports aggregation (Batch data processing ) and aggregate calculations using the native operations.

It is a schema less database.

It supports map reduce (Large volume of data into useful aggregated data.)

It also supports geospatial and secondary indexes on the collections.

Indexes are efficient execution of queries.Create a simple and complex indexes which will be used for speeding up the query execution.

Read More

What is Amazon ElasticCache ?

Many applications getting popular which is great, but when it comes to performance under heavy load, it is very bad. At the peak times, users are just giving up.
How we can solve this issue without re-engineering the application?

Amazon ElasticCache will solve the problem. It improves the performance of web applications by allowing the retrive the information fast, in-memory caches instead of relying on slow disk based databases.
It automates the time consuming tasks such as infrastructure provisioning, managing persistance,installing the software and patch management etc.

Amazon ElasticCache are managed in-memory service. You can define a cluster of nodes. You have the option for redis as well.
AWS will take care of the cluster management, in-memory cache etc.

Read More

Install Nodejs on Ubuntu 16.04

Node.js is a platform built on chrome's Javascript runtime for easily bulding fast and scalabale apps.
It uses an event driven, non blocking I/O model that makes it light weight and efficient for data intensive apps that run across distributed devices.

first update api package

$sudo apt-get update

Using apt package manager,   then   $sudo apt-get install nodejs

Install npm package 
$sudo apt-get install npm 

Read More

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

sudo dockerd \
      --storage-opt dm.datadev=/dev/sda1 \
      --storage-opt dm.metadatadev=/dev/sdf

INFO[0000] libcontainerd: previous instance of containerd still alive (1271)
INFO[0000] [graphdriver] using prior storage driver: aufs
Error starting daemon: error while opening volume store metadata database: timeout

$ systemctl daemon-reload
$ sudo service docker restart
$ sudo service docker status (should see active (running))
$ sudo docker run hello-world

Read More

Docker Machine

It is a tool thats lets you install Docker Engine on virtual hosts.

It creates servers, installs docker and then configures the docker client to take to them.


Linux :

ubuntu@ip-192.33.33.22:~$ docker-machine ls
docker-machine: command not found

Install docker machine

$ curl -L https://github.com/docker/machine/releases/download/v0.12.1/docker-machine-`uname -s`-`uname -m` >/tmp/docker-machine &&
chmod +x /tmp/docker-machine &&
sudo cp /tmp/docker-machine /usr/local/bin/docker-machine

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   617    0   617    0     0    428      0 --:--:--  0:00:01 --:--:--   428
100 25.1M  100 25.1M    0     0   490k      0  0:00:52  0:00:52 --:--:-- 1211k

ubuntu@ip-192.33.33.22:~$ docker-machine version
docker-machine version 0.12.1, build c8b17e8

Example :

$ docker-machine create -d digitalocean --digitalocean-access-token=secret dev

$ docker-machine ls

NAME      ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER   ERRORS

dev   -          digitalocean   Running   tcp://253.1.113.91:4366             v1.8.9

Read More

Install AWS Cli

Installing AWS Cli on Mac

Installing with Homebrew is easy. It is a popular package manager for MAC OSX.

To install Homebrew, 

Check , ruby is installed in your mac ,

ruby -v

Install homebrew 

Once homebrew installed, install python. To install Python, Gcc to be installed. you need xcode or commandline tools to be installed

you can download and install xcode or commandline tool.

https://developer.apple.com/download/more/

Step1 : brew install python

Install with Homebrew. It is a popular package manager for MAC OSX.

To install Homebrew, you must install xcode first.

Then install homebrew

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

$ brew doctor

to check brew installed properly.

Step2 :pip install awscli

Terminal output is here

Updating Homebrew...

==> Auto-updated Homebrew!

Updated 1 tap (homebrew/core).

==> New Formulae

apache-arrow        freedink            ipython@5           overmind

asdf                get_iplayer         jose                packetq

bitcoin             git-cinnabar        kubectx             pycodestyle

dcm2niix            gradle-completion   libmypaint          rustup-init

docker2aci          grakn               libnice             tectonic

dwarf               haste-client        libopusenc          tokei

erlang@19           iamy                mruby-cli           vips

faas-cli            insect              neovim              wpscan

fd                  ipython             openfortivpn        zork

==> Updated Formulae

abcl                                     libextractor

advancemame                              libgcrypt

afflib                                   libgit2

afl-fuzz                                 libgit2-glib

allure                                   libgosu

amazon-ecs-cli                           libgphoto2

      zurl

libepoxy

==> Renamed Formulae

osh -> etsh                              speedtest_cli -> speedtest-cli

==> Deleted Formulae

kibana@4.1          node@0.10           node@0.12           snescom




==> Installing dependencies for python: sqlite

==> Installing python dependency: sqlite

==> Downloading https://homebrew.bintray.com/bottles/sqlite-3.19.3.el_capitan.bo

######################################################################## 100.0%

==> Pouring sqlite-3.19.3.el_capitan.bottle.tar.gz

==> Using the sandbox

==> Caveats

This formula is keg-only, which means it was not symlinked into /usr/local,

because macOS provides an older sqlite3.




If you need to have this software first in your PATH run:

  echo 'export PATH="/usr/local/opt/sqlite/bin:$PATH"' >> ~/.bash_profile




For compilers to find this software you may need to set:

    LDFLAGS:  -L/usr/local/opt/sqlite/lib

    CPPFLAGS: -I/usr/local/opt/sqlite/include

For pkg-config to find this software you may need to set:

    PKG_CONFIG_PATH: /usr/local/opt/sqlite/lib/pkgconfig




==> Summary

🍺  /usr/local/Cellar/sqlite/3.19.3: 12 files, 3MB

==> Installing python

Warning: Building python from source:

  The bottle needs the Apple Command Line Tools to be installed.

  You can install them, if desired, with:

    xcode-select --install




==> Downloading https://www.python.org/ftp/python/2.7.13/Python-2.7.13.tar.xz

######################################################################## 100.0%

==> ./configure --prefix=/usr/local/Cellar/python/2.7.13 --enable-ipv6 --dataroo

==> make

==> make install PYTHONAPPSDIR=/usr/local/Cellar/python/2.7.13

==> make frameworkinstallextras PYTHONAPPSDIR=/usr/local/Cellar/python/2.7.13/sh

==> Downloading https://files.pythonhosted.org/packages/26/d1/dc7fe14ce4a3ff3fae

######################################################################## 100.0%

==> Downloading https://files.pythonhosted.org/packages/11/b6/abcb525026a4be042b

######################################################################## 100.0%

==> Downloading https://files.pythonhosted.org/packages/c9/1d/bd19e691fd4cfe908c

######################################################################## 100.0%

==> /usr/local/Cellar/python/2.7.13/bin/python -s setup.py --no-user-cfg install

==> /usr/local/Cellar/python/2.7.13/bin/python -s setup.py --no-user-cfg install

==> /usr/local/Cellar/python/2.7.13/bin/python -s setup.py --no-user-cfg install

==> Caveats

Pip and setuptools have been installed. To update them

  pip install --upgrade pip setuptools




You can install Python packages with

  pip install <package>




They will install into the site-package directory

  /usr/local/lib/python2.7/site-packages




See: http://docs.brew.sh/Homebrew-and-Python.html

==> Summary

🍺  /usr/local/Cellar/python/2.7.13: 6,337 files, 86.7MB, built in 6 minutes 19 seconds
pip install awscli

Collecting awscli

  Downloading awscli-1.11.120-py2.py3-none-any.whl (1.2MB)

    100% |████████████████████████████████| 1.2MB 79kB/s 

Collecting botocore==1.5.83 (from awscli)

  Downloading botocore-1.5.83-py2.py3-none-any.whl (3.5MB)

    100% |████████████████████████████████| 3.6MB 23kB/s 

Collecting rsa<=3.5.0,>=3.1.2 (from awscli)

  Downloading rsa-3.4.2-py2.py3-none-any.whl (46kB)

    100% |████████████████████████████████| 51kB 43kB/s 

Collecting docutils>=0.10 (from awscli)

  Downloading docutils-0.13.1-py2-none-any.whl (537kB)

    100% |████████████████████████████████| 542kB 72kB/s 

Collecting PyYAML<=3.12,>=3.10 (from awscli)

  Downloading PyYAML-3.12.tar.gz (253kB)

    100% |████████████████████████████████| 256kB 181kB/s 

Collecting colorama<=0.3.7,>=0.2.5 (from awscli)

  Downloading colorama-0.3.7-py2.py3-none-any.whl

Collecting s3transfer<0.2.0,>=0.1.9 (from awscli)

  Downloading s3transfer-0.1.10-py2.py3-none-any.whl (54kB)

    100% |████████████████████████████████| 61kB 28kB/s 



  Running setup.py bdist_wheel for PyYAML ... done

  Stored in directory: /Users/srinivas/Library/Caches/pip/wheels/2c/f7/79/13f3a12cd723892437c0cfbde1230ab4d82947ff7b3839a4fc

Successfully built PyYAML

Installing collected packages: six, python-dateutil, docutils, jmespath, botocore, pyasn1, rsa, PyYAML, colorama, futures, s3transfer, awscli

Successfully installed PyYAML-3.12 awscli-1.11.120 botocore-1.5.83 colorama-0.3.7 docutils-0.13.1 futures-3.1.1 jmespath-0.9.3 pyasn1-0.2.3 python-dateutil-2.6.1 rsa-3.4.2 s3transfer-0.1.10 six-1.10.0

Read More