AWS S3 Tutorial

Amazon Simple Storage Service is also known as Amazon S3. It is highly scalable object storage in the cloud securely.
It is used to store any amount of data.
MultiPart form uploading to AWS

S3 is Object Storage with a simple web service interface to store and retrive any amount of data from anywhere on the web.

What is multi part upload ?
1. Initiate the Multi part Upload
2. Seperate the object into multiple parts.
3. Upload the parts in any order, one at a time or in the parallel.

Date transmission - Browser going to App server and going to S3.
https://s3.amazonaws.com/upload-demo-images/5e64truyt47653745tr46tr736t4rt4r3t46

Nodejs code :
Install the dependencies.
npm install s3-stream-upload


var UploadStream = require("s3-stream-upload");
var S3 = require("aws-sdk").S3;

var key = "file.mp3";
var s3 = new S3();

fs.readFileStream(__dirname + "/file.mp3")
  .pipe(UploadStream(s3, { Bucket: "my-S3-bucket", Key: key }))
  .on("error", function (err) {
    console.error(err);
  })
  .on("finish", function () {
    console.log("File uploaded!");
  });

my sample s3 url - 'https://s3.amazonaws.com/upload-demo-images/';

Read More

Aws Security Groups

The very first thing to ensure that the unwanted or unauthorized traffic doesn't get in into the instances using Security Group.

What is a Security Group?
It is a AWS firewall solution which filters the incoming and outgoing traffic from an instance.
Filtering is done based upon the ip protocols,ports and source/destination ip addresses.

At the backend requires X.509 certificate and key to authorise changes.

How to configure security groups while creating instance.
You can configure security groups while creating instance or anytime after creating instance.

Figure 1.0

Figure 1.1

Here you can configure security group.

Start configure security rules. Security rules for set of firewall rules that control the traffic. what kind of traffic can reach the instance.

The rules we are going to configure are Inbound rules.

SSH/RDP:

If we use linux based machine, configure SSH. SSH stands for secure shell. Secure access to the remote machine.It always communicates over port 22.It is static and you cannot change the port.

If it is Windows based machine, configure RDP. By default it will give root access to the machine.

If we select source anywhere, across internet can access the machine via SSH. This is quite dangerous. 

You can choose custom ip for security purpose. 

Web Traffic:

Web traffic can come from any source.

0.0.0.0/0     -> allows all the traffic.which is not secure.

HTTP communicates over port 80.

HTTPS always communicates over port 443. 

Depends on what traffic you want to allow in, you can add rule here.

Figure 1.2

Figure 1.3

Read More

aws instance types

General Purpose (T2, M3, M4 class):

These provides overall balance of Ram, CPU and provide burstable performance.
IOP starts from 100 and upto 3000.

Compute Optimized (C3 and C4 class):

It is meant for hude traffic.

Storage Optimized(I3  and D2 class):

It is for large datawarehouse.

Memory Optimized(X1, R3 and R4 class):

It is for Memcached and enterprise apps.

Accelerated Computing(P2, G3 and F1 class):

It is for general-purpose GPU compute applications,graphics-intensive applications and FPGAs.

Read More

Unabme to run docker without sudo

I am unable to run docker without sudo.

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.25/containers/json: dial unix /var/run/docker.sock: connect: permission denied


I added docker to user group

sudo usermod -aG docker $USER

Still i am unable to run without sudo.

Read More

docker Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Warning: failed to get default registry endpoint from daemon (Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?). Using system default: https://index.docker.io/v1/

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Read More

Jupyterhub is not running on ubuntu 16.04

$ jupyterhub

/home/ubuntu/jupyterhub_config.py:6: UserWarning: IPython.utils.localinterfaces has moved to jupyter_client.localinterfaces

  from IPython.utils.localinterfaces import public_ips

[I 2017-08-07 18:36:18.545 JupyterHub app:724] Loading cookie_secret from /home/ubuntu/jupyterhub_cookie_secret

[W 2017-08-07 18:36:18.576 JupyterHub app:365] 

    Generating CONFIGPROXY_AUTH_TOKEN. Restarting the Hub will require restarting the proxy.

    Set CONFIGPROXY_AUTH_TOKEN env or JupyterHub.proxy_auth_token config to avoid this message.

    

[W 2017-08-07 18:36:18.579 JupyterHub app:864] No admin users, admin interface will be unavailable.

[W 2017-08-07 18:36:18.580 JupyterHub app:865] Add any administrative users to `c.Authenticator.admin_users` in config.

[I 2017-08-07 18:36:18.580 JupyterHub app:892] Not using whitelist. Any authenticated user will be allowed.

[I 2017-08-07 18:36:18.593 JupyterHub app:1453] Hub API listening on http://172.31.28.236:8081/hub/

[W 2017-08-07 18:36:18.596 JupyterHub app:1174] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...

[I 2017-08-07 18:36:18.596 JupyterHub app:1176] Starting proxy @ http://*:8000/

/usr/bin/env: ‘node’: No such file or directory

[C 2017-08-07 18:36:19.605 JupyterHub app:1459] Failed to start proxy

    Traceback (most recent call last):

      File "/home/ubuntu/miniconda3/lib/python3.6/site-packages/jupyterhub/app.py", line 1457, in start

        yield self.start_proxy()

      File "/home/ubuntu/miniconda3/lib/python3.6/site-packages/jupyterhub/app.py", line 1197, in start_proxy

        _check()

      File "/home/ubuntu/miniconda3/lib/python3.6/site-packages/jupyterhub/app.py", line 1193, in _check

        raise e

    RuntimeError: Proxy failed to start with exit code 127

solution:

sudo apt-get install npm nodejs-legacy

Read More

Install NodeJS on Ubuntu 16.04

1.  sudo apt-get update

2.  sudo apt-get install nodejs

3.  sudo apt-get install npm

 sudo apt-get install nodejs

Reading package lists... Done

Building dependency tree       

Reading state information... Done

The following additional packages will be installed:

  libuv1

The following NEW packages will be installed:

  libuv1 nodejs

0 upgraded, 2 newly installed, 0 to remove and 23 not upgraded.

Need to get 3218 kB of archives.

After this operation, 13.4 MB of additional disk space will be used.

Do you want to continue? [Y/n] Y

Get:1 http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu xenial/universe amd64 libuv1 amd64 1.8.0-1 [57.4 kB]

Get:2 http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 nodejs amd64 4.2.6~dfsg-1ubuntu4.1 [3161 kB]

Fetched 3218 kB in 0s (61.6 MB/s)

perl: warning: Setting locale failed.

perl: warning: Please check that your locale settings:

LANGUAGE = (unset),

LC_ALL = (unset),

LC_CTYPE = "UTF-8",

LANG = "en_US.UTF-8"

    are supported and installed on your system.

perl: warning: Falling back to a fallback locale ("en_US.UTF-8").

locale: Cannot set LC_CTYPE to default locale: No such file or directory

locale: Cannot set LC_ALL to default locale: No such file or directory

Selecting previously unselected package libuv1:amd64.

(Reading database ... 51035 files and directories currently installed.)

Preparing to unpack .../libuv1_1.8.0-1_amd64.deb ...

Unpacking libuv1:amd64 (1.8.0-1) ...

Selecting previously unselected package nodejs.

Preparing to unpack .../nodejs_4.2.6~dfsg-1ubuntu4.1_amd64.deb ...

Unpacking nodejs (4.2.6~dfsg-1ubuntu4.1) ...

Processing triggers for libc-bin (2.23-0ubuntu9) ...

Processing triggers for man-db (2.7.5-1) ...

Setting up libuv1:amd64 (1.8.0-1) ...

Setting up nodejs (4.2.6~dfsg-1ubuntu4.1) ...

update-alternatives: using /usr/bin/nodejs to provide /usr/bin/js (js) in auto mode

Processing triggers for libc-bin (2.23-0ubuntu9) ...

ubuntu@ip-192-31-28-436:~$ sudo apt-get install npm

Reading package lists... Done

Building dependency tree       

Reading state information... Done

The following additional packages will be installed:

  binutils build-essential cpp cpp-5 dpkg-dev fakeroot g++ g++-5 gcc gcc-5 gyp

  javascript-common libalgorithm-diff-perl libalgorithm-diff-xs-perl

  libalgorithm-merge-perl libasan2 libatomic1 libc-dev-bin libc6-dev libcc1-0

  libcilkrts5 libdpkg-perl libfakeroot libfile-fcntllock-perl libgcc-5-dev

  libgomp1 libisl15 libitm1 libjs-inherits libjs-jquery libjs-node-uuid

  libjs-underscore liblsan0 libmpc3 libmpx0 libpython-stdlib

  libpython2.7-minimal libpython2.7-stdlib libquadmath0 libssl-dev libssl-doc

  libstdc++-5-dev libtsan0 libubsan0 libuv1-dev linux-libc-dev make

  manpages-dev node-abbrev node-ansi node-ansi-color-table node-archy

  node-async node-block-stream node-combined-stream node-cookie-jar

  node-delayed-stream node-forever-agent node-form-data node-fstream

  node-fstream-ignore node-github-url-from-git node-glob node-graceful-fs

  node-gyp node-inherits node-ini node-json-stringify-safe node-lockfile

  node-lru-cache node-mime node-minimatch node-mkdirp node-mute-stream

  node-node-uuid node-nopt node-normalize-package-data node-npmlog node-once

  node-osenv node-qs node-read node-read-package-json node-request node-retry

  node-rimraf node-semver node-sha node-sigmund node-slide node-tar

  node-tunnel-agent node-underscore node-which nodejs-dev python

  python-minimal python-pkg-resources python2.7 python2.7-minimal zlib1g-dev

Suggested packages:

  binutils-doc cpp-doc gcc-5-locales debian-keyring g++-multilib

  g++-5-multilib gcc-5-doc libstdc++6-5-dbg gcc-multilib autoconf automake

  libtool flex bison gdb gcc-doc gcc-5-multilib libgcc1-dbg libgomp1-dbg

  libitm1-dbg libatomic1-dbg libasan2-dbg liblsan0-dbg libtsan0-dbg

  libubsan0-dbg libcilkrts5-dbg libmpx0-dbg libquadmath0-dbg apache2

  | lighttpd | httpd glibc-doc libstdc++-5-doc make-doc node-hawk

  node-aws-sign node-oauth-sign node-http-signature debhelper python-doc

  python-tk python-setuptools python2.7-doc binfmt-support

The following NEW packages will be installed:

  binutils build-essential cpp cpp-5 dpkg-dev fakeroot g++ g++-5 gcc gcc-5 gyp

  javascript-common libalgorithm-diff-perl libalgorithm-diff-xs-perl

  node-gyp node-inherits node-ini node-json-stringify-safe node-lockfile

  node-lru-cache node-mime node-minimatch node-mkdirp node-mute-stream

  node-node-uuid node-nopt node-normalize-package-data node-npmlog node-once

  node-osenv node-qs node-read node-read-package-json node-request node-retry

  node-rimraf node-semver node-sha node-sigmund node-slide node-tar

  node-tunnel-agent node-underscore node-which nodejs-dev npm python

  python-minimal python-pkg-resources python2.7 python2.7-minimal zlib1g-dev

0 upgraded, 102 newly installed, 0 to remove and 23 not upgraded.

Need to get 47.7 MB of archives.

After this operation, 188 MB of additional disk space will be used.

Do you want to continue? [Y/n] Y

Get:1 http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 libpython2.7-minimal amd64 2.7.12-1ubuntu0~16.04.1 [339 kB]

Get:2 http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 python2.7-minimal 

Unpacking node-npmlog (0.0.4-1) ...

Selecting previously unselected package node-osenv.

Preparing to unpack .../node-osenv_0.1.0-1_all.deb ...

Unpacking node-osenv (0.1.0-1) ...

Selecting previously unselected package node-tunnel-agent.

Preparing to unpack .../node-tunnel-agent_0.3.1-1_all.deb ...

Unpacking node-tunnel-agent (0.3.1-1) ...

Selecting previously unselected package node-json-stringify-safe.

Preparing to unpack .../node-json-stringify-safe_5.0.0-1_all.deb ...

Unpacking node-json-stringify-safe (5.0.0-1) ...

Selecting previously unselected package node-qs.

Preparing to unpack .../node-qs_2.2.4-1_all.deb ...

Unpacking node-qs (2.2.4-1) ...

Selecting previously unselected package node-request.

Preparing to unpack .../node-request_2.26.1-1_all.deb ...

Unpacking node-request (2.26.1-1) ...

Selecting previously unselected package node-semver.

Preparing to unpack .../node-semver_2.1.0-2_all.deb ...

Unpacking node-semver (2.1.0-2) ...

Selecting previously unselected package node-tar.

Preparing to unpack .../node-tar_1.0.3-2_all.deb ...

Unpacking node-tar (1.0.3-2) ...

Selecting previously unselected package node-which.

Preparing to unpack .../node-which_1.0.5-2_all.deb ...

Unpacking node-which (1.0.5-2) ...

Selecting previously unselected package node-gyp.

Preparing to unpack .../node-gyp_3.0.3-2ubuntu1_all.deb ...

Unpacking node-gyp (3.0.3-2ubuntu1) ...

Selecting previously unselected package node-ini.

Preparing to unpack .../node-ini_1.1.0-1_all.deb ...

Unpacking node-ini (1.1.0-1) ...

Selecting previously unselected package node-lockfile.

Preparing to unpack .../node-lockfile_0.4.1-1_all.deb ...

Unpacking node-lockfile (0.4.1-1) ...

Selecting previously unselected package node-mute-stream.

Preparing to unpack .../node-mute-stream_0.0.4-1_all.deb ...

Unpacking node-mute-stream (0.0.4-1) ...

Selecting previously unselected package node-normalize-package-data.

Preparing to unpack .../node-normalize-package-data_0.2.2-1_all.deb ...

Unpacking node-normalize-package-data (0.2.2-1) ...

Selecting previously unselected package node-read.

Preparing to unpack .../node-read_1.0.5-1_all.deb ...

Unpacking node-read (1.0.5-1) ...

Selecting previously unselected package node-read-package-json.

Preparing to unpack .../node-read-package-json_1.2.4-1_all.deb ...

Unpacking node-read-package-json (1.2.4-1) ...

Selecting previously unselected package node-retry.

Preparing to unpack .../node-retry_0.6.0-1_all.deb ...

Unpacking node-retry (0.6.0-1) ...

Setting up node-lru-cache (2.3.1-1) ...

Setting up node-sigmund (1.0.0-1) ...

Setting up node-minimatch (1.0.0-1) ...

Setting up node-fstream-ignore (0.0.6-2) ...

Setting up node-github-url-from-git (1.1.1-1) ...

Setting up node-once (1.1.1-1) ...

Setting up node-glob (4.0.5-1) ...

Setting up nodejs-dev (4.2.6~dfsg-1ubuntu4.1) ...

Setting up node-nopt (3.0.1-1) ...

Setting up node-npmlog (0.0.4-1) ...

Setting up node-osenv (0.1.0-1) ...

Setting up node-tunnel-agent (0.3.1-1) ...

Setting up node-json-stringify-safe (5.0.0-1) ...

Setting up node-qs (2.2.4-1) ...

Setting up node-request (2.26.1-1) ...

Setting up node-semver (2.1.0-2) ...

Setting up node-tar (1.0.3-2) ...

Setting up node-which (1.0.5-2) ...

Setting up node-gyp (3.0.3-2ubuntu1) ...

Setting up node-ini (1.1.0-1) ...

Setting up node-lockfile (0.4.1-1) ...

Setting up node-mute-stream (0.0.4-1) ...

Setting up node-normalize-package-data (0.2.2-1) ...

Setting up node-read (1.0.5-1) ...

Setting up node-read-package-json (1.2.4-1) ...

Setting up node-retry (0.6.0-1) ...

Setting up node-sha (1.2.3-1) ...

Setting up node-slide (1.1.4-1) ...

Setting up npm (3.5.2-0ubuntu4) ...

Processing triggers for libc-bin (2.23-0ubuntu9) ...

Read More