Sunday, December 21, 2014

How Proxy & NAT works ?


NAT is an acronym for "Network Address Translation." Traditionally NAT is done by routers and firewalls. On the vast majority of networks the IP addresses given to computers are not publicly routable. This means that computers which share a network can talk to each other directly; however to talk to the Internet their local IP addresses must be "translated," to a publicly routable address. Traditionally this is done by a router or firewall. The router/firewall has one or more publicly routable addresses, which the whole Internet can direct communications to. It also has a local address on the inside network, a network which it shares with its PCs and other devices. When a PC makes a request out to the Internet, it's directed to the router/firewall. The router/firewall changes the source IP address of the packets to its public IP address, notes the connection request in its memory, and sends them on their way on the Internet. When a response is received, the router looks up the connection in its memory, and this time changes the destination address from the public IP to the local IP of the machine which initiated communications. It then sends the packet on its way on the inside network. On Linux this is referred to as "source" NAT. Cisco calls it "dynamic" NAT.

When you have a server behind a firewall, "destination" or "static" NAT is performed. Connection requests come into public IP(s) on the firewall or router. The router/firewall looks in its memory and determines if the port is supposed to be NATed to an inside server. If it is, the router/firewall changes the destination address from the public IP it arrived with to the corresponding private IP of the corresponding server in its memory. It then sends the request onto the inside network. There is also 1:1 NAT, which Cisco refers to as a type of static NAT. In 1:1 static NAT, an IP on one interface is directly mapped to a corresponding IP on another. A good and properly configured firewall will still inspect and filter the traffic being translated. There are more complicated scenarios involving NAT, but these are the three most common and basic.

Note that NAT requires the traffic to be running through the router/firewall. It affects traffic as it moves from one routed interface to another. Applications are generally not aware that NAT is taking place, and there is no client side configuration.

There are several different kinds of proxy, each adapted for specific use cases.

Forward proxies are run by servers. To use one client applications must be configured to use the proxy. They are mostly used in two cases. The first is in the corporate world, where they can be used to cache and filter. When a properly configured client application wants to initiate communications, it makes the request to the proxy. The proxy can then determine whether the connection is allowed. This is part of why corporations use them, to enforce Internet access and data security policies. If the connection is allowed, the proxy server makes a connection to the requested resource and sends it to the client who requested it.

Forward proxies can also cache. So, and this was more true in the dial up days, if I have 100 workers who all browse to the same web page to perform their jobs, the proxy can download it once, and send the same copy to subsequent requesters. This would cut down on bandwidth. Another use of traditional forward proxies is to have your traffic appear to come from somewhere else. Remember, the proxy makes its own connection, from its own IP, to the destination resource, and marshals the communications back and forth. The proxy clients requests appear to come from the proxy itself, and not client. Forward proxies can be used across the Internet, even if the traffic wouldn't normally take that route. This also helps to differentiate them from NAT.

Transparent proxies are similar to forward proxies, but the client applications aren't aware of them. They are newer to the block than traditional forward proxies; as processors and RAM have gotten cheaper, the use of transparent and caching proxies on routers and firewalls instead of forward proxy servers is on the rise. The router/firewall sends all or some requests (generally HTTP requests,) passing through it to a software proxy application it's running, such as squid, before they are sent out on the Internet. This gives squid a chance to either deny the connection or serve it from its own cache. The upside to these is that you don't have to configure proxy server settings on individual PCs or through Group Policy.

A reverse proxy is used to load balance and protect inside web servers from outside hosts. So, if I'm running a website on an insecure platform, I can setup an Apache reverse proxy and use that to allow access from the Internet to my web servers. I can then setup Apache in a secure manner and protect my IIS servers. Cisco makes the ACE which allows high end switches and routers to run reverse proxies.

Thursday, December 18, 2014

Docker Installation and working in Windows

The Docker Engine uses Linux-specific kernel features, so to run it on Windows we need to use a lightweight virtual machine (vm). You use the Windows Docker client to control the virtualized Docker Engine to build, run, and manage Docker containers.
To make this process easier, we've designed a helper application called Boot2Docker that installs the virtual machine and runs the Docker daemon.

Installation

  1. Download the latest release of the Docker for Windows Installer
  2. Run the installer, which will install VirtualBox, MSYS-git, the boot2docker Linux ISO, and the Boot2Docker management tool.
  3. Run the Boot2Docker Start shell script from your Desktop or Program Files > Boot2Docker for Windows. The Start script will ask you to enter an ssh key passphrase - the simplest (but least secure) is to just hit [Enter].

The Boot2Docker Start script will connect you to a shell session in the virtual machine. If needed, it will initialize a new VM and start it.

Upgrading

  1. Download the latest release of the Docker for Windows Installer
  2. Run the installer, which will update the Boot2Docker management tool.
  3. To upgrade your existing virtual machine, open a terminal and run:
    boot2docker stop
    boot2docker download
    boot2docker start

Running Docker

Boot2Docker will log you in automatically so you can start using Docker right away.
Let's try the hello-world example image. Run
$ docker run hello-world
This should download the very small hello-world image and print a Hello from Docker. message.

Further Details

The Boot2Docker management tool provides several commands:
$ ./boot2docker
Usage: ./boot2docker [<options>] {help|init|up|ssh|save|down|poweroff|reset|restart|config|status|info|ip|delete|download|version} [<args>]

Container port redirection

If you are curious, the username for the boot2docker default user is docker and the password is tcuser.
The latest version of boot2docker sets up a host only network adaptor which provides access to the container's ports.
If you run a container with an exposed port:
docker run --rm -i -t -p 80:80 nginx
Then you should be able to access that nginx server using the IP address reported to you using:
boot2docker ip
Typically, it is 192.168.59.103, but it could get changed by Virtualbox's DHCP implementation.
For further information or to report issues, please see the Boot2Docker site

Monday, December 15, 2014

Deploy ROOT.war for static content in JbossAS 7

The JBossAS7 root is served from the static $JBOSS_HOME/welcome-content directory (not a servlet app).

You can still add the ROOT.war into $JBOSS_HOME/standalone/deployments directory, but you must first disable the above first. Go in your "$JBOSS_HOME/standalone/configuration/standalone.xml", then find

<subsystem xmlns="urn:jboss:domain:web:1.1"

and in the section find and set "enable-welcome-root="false" instead of "true".

With that disabled, your ROOT.war should work just as the old days.

If your ROOT.war is a directory (exploded war mode), then you need to create an empty file name "ROOT.war.dodeploy" to tell JBossAS to deploy it. I think JBossAS will only auto deploy packaged single .war file only.

Saturday, December 13, 2014

System calls for Files

System call OPEN


Opening or creating a file can be done using the system call open. The syntax is:

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
int open(const char *path,
   int flags,... /* mode_t mod */);

This function returns the file descriptor or in case of an error -1. The number of arguments that this function can have is two or three. The third argument is used only when creating a new file. When we want to open an existing file only two arguments are used. The function returns the smallest available file descriptor. This can be used in the following system calls: read, write, lseek and close. The effective UID or the effective GID of the process that executes the call has to have read/write rights, based on the value of the argument flags. The file pointer is places on the first byte in the file. The argument flags is formed by a bitwise OR operation made on the constants defined in the fcntl.h header.

System call CREATE:


A new file can be created by:

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
int creat(const char *path, mode_t mod);

The function returns the file descriptor or in case of an error it returns the value -1. This call is equivalent with:

     open(path, O_WRONLY | O_CREAT | O_TRUNC, mod);

The argument path specifies the name of the file, while mod defines the access rights. If the created file doesn’t exist, a new i-node is allocated and a link is made to this file from the directory it was created in. The owner of the process that executes the call - given by the effective UID and the effective GUID - must have writing permission in the directory. The open file will have the access rights that were specified in the second argument (see umask, too). The call returns the smallest file descriptor available. The file is opened for writing and its initial size is 0. The access time and the modification time are updated in the i-node. If the file exists (permission to search the directory is needed), it looses its contents and it will be opened for writing. The ownership and the access permissions won’t be modified. The second argument is ignored.



System call READ


When we want to read a certain number of bytes starting from the current position in a file, we use the read call. The syntax is:

#include <unistd.h>
ssize_t read(int fd, void* buf, size_t noct);

The function returns the number of bytes read, 0 for end of file (EOF) and -1 in case an error occurred. It reads noct bytes from the open file referred by the fd descriptor and it puts it into a buffer buf. The pointer (current position) is incremented automatically after a reading that certain amount of bytes. The process that executes a read operation waits until the system puts the data from the disk into the buffer.

System call WRITE


For writing a certain number of bytes into a file starting from the current position we use the write call. Its syntax is:

#include <unistd.h>
ssize_t write(int fd, const void* buf, size_t noct);

The function returns the number of bytes written and the value -1 in case of an error. It writes noct bytes from the buffer buf into the file that has as its descriptor fd. It is interesting to note that the actual writing onto the disk is delayed. This is done at the initiative of the root, without informing the user when it is done. If the process that did the call or an other process reads the data that haven’t been written on the disk yet, the system reads all this data out from the cache buffers. The delayed writing is faster, but it has three disadvantages:
a)      a disk error or a system error may cause loosing all the data
b)      a process that had the initiative of a write operation cannot be informed in case a writing error occurred
c)      the physical order of the write operations cannot be controlled.
To eliminate these disadvantages, in some cases the O_SYNC is used. But as this slows down the system and considering the reliability of today’s systems it is better to use the mechanism which includes using cache buffers. 

System call CLOSE


For closing a file and thus eliminating the assigned descriptor we use the system call close.

#include <unistd.h>
int close(int fd);

The function returns 0 in case of success and -1 in case of an error. At the termination of a process an open file is closed anyway.

System call LINK


To link an existing file to another directory (or to the same directory) link can be used. To make such a link in fact means to set a new name or a path to an existing file. The link system call creates a hard link. Creating symbolic links can be done using symlink system call. The syntax of link is:

#include <unistd.h>
int link(const char* oldpath, const char* newpath);
int symlink(const char* oldpath, const char* newpath);

The function returns 0 in case of success and -1 in case of an error. The argument oldpath has to be a path to an existing file. Only the root has the right to set a link to a directory.

System call UNLINK


To delete a link (a path) in a directory we can use the unlink system call. Its syntax is:
           
#include <unistd.h>
int unlink(const char* path);

The function returns 0 in case of success and -1 otherwise. The function decrements the hard link counter in the i-node and deletes the appropriate directory entry for the file whose link was deleted. If the number of links of a file becomes 0 then the space occupied by the file and its i-node will be freed. Only the root can delete a directory. 

System calls STAT, LSTAT and FSTAT


In order to obtain more details about a file the following system calls can be used: stat, lstat or fstat.

#include <sys/types.h>
#include <sys/stat.h>
int stat(const char* path, struct stat* buf);
int lstat(const char* path, struct stat* buf);
int fstat(int df, struct stat* buf);

These three functions return 0 in case of success and -1 in case of an error. The first two gets as input parameter a name of a file and completes the structure of the buffer with additional information read from its i-node. The fstat function is similar, but it works for files that were already opened and for which the file descriptor is known. The difference between stat and lstat is that in case of a symbolic link, function stat returns information about the linked (refered) file, while lstat returns information about the symbolic link file.


System call CHMOD


To modify the access rights for an existing file we use:

#include <sys/types.h>
#include <sys/stat.h>
int chmod(const char* path, mode_t mod);

The function returns 0 in case of a success and -1 otherwise. The chmod call modifies the access rights of the file specified by the path depending on the access rights specified by the mod argument. To be able to modify the access rights the effective UID of the process has to be identical to the owner of the file or the process must have root rights.

The mod argument can be specified by one of the symbolic constants defined in the sys/stat.h header. 

System call CHOWN


This system call is used to modify the owner (UID) and the group (GID) that a certain file belongs to. The syntax of the function is:

#include <sys/types.h>
#include <unistd.h>
int chown(const char* path, uid_t owner, gid_t grp);

The function returns 0 in case of success and -1 in case of an error. Calling this function will change the owner and the group of the file specified by the argument path to the values specified by the arguments owner and grp. None of the users can change the owner of any file (even of his/her own files), except the root user, but they can change the GID for their own files to that of any group they belong to.

 System call mount /Umount:

#include <sys/mount.h> 

int mount(const char *source, const char *target, 
          const char *filesystemtype, unsigned long mountflags, 
          const void *data); 

int umount(const char *target);
int umount2(const char *target, int flags);

Description:
 
mount() attaches the filesystem specified by source (which is often a device name, but can also be a directory name or a dummy) to the directory specified by target. umount() and umount2() remove the attachment of the (topmost) filesystem mounted on target.




 

General overview of the system (unix)

General overview of the system

Benefits of Unix

  • The system is written in high level language “C” which makes it easier to read understand and update. It off-course is little slower in comparison to what written in assembly language but the advantages are far greater and possibilities are endless.
  • Complex programs can be built on simple programs on unix systems.
  • It has hierarchal file system which allows easy maintenance and efficient implementation.
  • It has consistent format for files. The files are stored as byte streams which makes it easier for the programs to use files.
  • It provides simple and consistent interface to the peripheral devices.
  • It completely hides the machine architecture from the user which makes it easier for the programmers to write programs independent of hardware.

System Structure

The unix system can be viewed as the set of layers. The lowermost layer is the hardware layer which is not the part of the unix operating system. The operating system is called the system kernel or the kernel.
Kernel is the layer where the actual operating system code and functionality resides. It is in complete isolation from the user programs. This makes it easier for the programs to be ported onto other system provided the kernel are same.
If a user program want to perform any task it can do so by talking to the kernel. The programs interact with the kernel by using the system calls. The system calls instruct the kernel to do various operations.
Other user programs can be built on top of the lower level programs using these lower level programs and system calls.

User perspective – The file system

The characteristics of unix file system are
  • A hierarchal structure.
  • Consistent treatment of data
  • Ability to create and delete files
  • Dynamic growth of files
  • Peripheral devices are also treated as files
The file system is organized as a tree. The root node is called “root” and is denoted by “/”. Every non leaf node in this structure is a directory and every leaf node is a file/special device file.
The name of the file is given by the path name.
A full path name starts with the root directory i.e. a slash character and specifies the file that can be found by travestying the tree. Some examples of paths could be “/etc/passwd”,  “/bin/who” and “/usr/src/programs/test.c”.

The path that starts from the root directory is called the absolute path. Alternatively we can give path of any file relative to any other directory. This path will be called relative path.

The files are just stream of bytes it is up-to the program to interpret these bytes. Directories are also files i.e. a stream of bytes but the operating system program knows how to interpret them as directories. Example program could be “ls”

Permission to any file is governed by the file access permissions. Access permissions are set independently for read, write and execute. These permissions are set independently for the file owner, file group and everyone else. Access permission looks like
rwx-rwx-rwx  (We will see more of this in later chapters)

Unix treats devices as if they are files. Every device is treated as special files and occupy position in the file system. Programs can access devices using the same syntax as if they were accessing files. Syntax of reading and writing on devices is more or less same as reading and writing regular files. Devices are protected in the same way as files i.e. using access pemissions.

User perspective – Processing environment

A source code is our program source code, an executable file is the program for our source code and the process is the instance of our program in execution. Many processes can execute simultaneously in uix. (Multiprogramming or multitasking). Also many instances of one program can run simultaneously. Each instance of this program is one process. Various system calls allows control of the state of the process. The state of a process indicates its status at a particular time. The process state could by any one of the following.
Process state information along with other useful information is stored in a process control block. Every process has its own process control block or PCB.
Unix shell allows three types of commands.
  1. An executable file created by compilation of our source code.
  2. An executable command that contains a sequence of shell commands.
  3. An internal shell command.
The shell, usually, run the commands synchronously. However these commands can also be run asynchronously.

User perspective – building block primitives

Unix allows user to write small programs, in a modular way. These programs can be used as building blocks to build the complex programs.
Unix has three standard files:
  1. Standard input file
  2. Standard output file
  3. Standard error file
Typically when we run shell our terminal (monitor) is serving as these three files. (remember devices can be treated as files).

One primitive building block available to the shell user is the redirect I/O. for example
ls

this command list down all the files in the current directory.
ls > output

this command will send this list of files to a file named “output” instead of the terminal.

The second building block primitive is the PIPE. Pipe allows a stream of data to be passed from processes. There is one reader process and one writer process.

ls | more

Operating system Services

The kernel layer provides various oerations on behalf of user processes. Some of the main services provided by the operating systems kernel are:
  • Process control: controlling the creating, termination and suspension of processes.
  • Scheduling processes: Since many programs can execute simultaneously in unix the process scheduling is also done by the kernel.
  • Main memory management: allocating main memory to the user programs and protecting the memory region where kernel is running. Also, protecting the memory region of one process from another process.
  • Virtual memory: managing the swap device and handling the swapping system. Controlling the pages in the paging system(memory allocation)
  • Secondary memory management: Managing the secondary storage for the efficient and timely retrieval and storage of data.
  • Peripheral devices: kernel controls the peripheral devices such as terminals, disk drives and network devices.

Assumptions about the hardware

When a process executes on unix it executes on two levels or we can say it executes in two modes.
  • User level
  • Kernel level
Processes in user mode can assess their own instructions but not the kernel instructions or the instruction of other processes. On the other hand processes in kernel mode can access kernel data and instructions as well as user data and instructions.

The system calls can only be executed in the kernel mode. If a user process running in user mode make a system calls the process shifts from user mode to kernel mode and then the kernel services the request and the system comes back to the user mode after the request is serviced.

Interrupts and exceptions

The devices can interrupt the CPU anytime asynchronously. On receiving the interrupt the kernel saves its current context(whatever it was doing) and jumps to service that interrupt. After the kernel is done servicing the interrupt it reloads its context and resumes whatever it was doing.

There might be a possibility that kernel is servicing one interrupt and another interrupt may occur. So whether or not to service that interrupt and stop whatever kernel was doing is decided by the interrupt priority(or interrupt levels). If a high priority interrupt occurs the kernel stops the previous one and jumps onto second. If a lower priority interrupt occurs the kernel will not stop what it was doing and that interrupt will have to wait. In other words the lower priority interrupt is blocked if kernel is servicing some high priority interrupt.

An exception occurs when a process does something unexpected. Exceptions are different from interrupts they occur as events. If an interrupt occurs in the middle of instruction, that instruction will be restarted after handling the exception. If the exception is not caused by the instruction but because of some other reasons and between two instructions then the next instruction is processed after handling the exception.

Memory management

The unix kernel is in main memory and the user programs are also in main memory so The operating system resides in the lower memory. User processes execute in the higher memory. There is always a possibility that user processes may try to access the lower memory either accidentally or intentionally thereby causing loss of operating system code and data. to prevent the user programs to corrupt the kernel memory the memory management is also done by the unix kernel.

Introduction to Kernel (unix)

kernel architecture

Introduction to system concepts – overview of file subsystem

The internal representation of the file is in the form of iNode. This inode contains the information about the file such as its layout on the disk, its owner, its access permissions and last accessed time.
This inode is short form for index node. Every file has one inode.  The inodes of all the files on the system are stored in inode table. When we create a new file a new entry in the inode table is created.
The kernel contain two data structures file table and user file descriptor table. The file table is global table at the kernel level but the user file descriptor table s for every process. When a process creates a file or opens a file the entry for that is made in both the tables.
The information about the current state of the file is maintained in the file table. For example if the file is being written the information about the current cursor position is kept in the file table. This file table also checks whether the accessing process has access to that file or not.
The user file descriptor table keeps a track of all the files opened by the processes and what are the relationships between these files.
file descriptors
The regular files and folders are kept on block devices like disks and tape drives. The drive has logical block numbers and physical block numbers and the mapping from logical to physical block numbers is done by disk driver.

File system layout

File system layout
  • The boot block occupies the beginning of the file system. This contains the bootstrap code that is required for the machine to boot.
  • Super block describes the state of the file system i.e. its size, maximum number of files that can be stored and the free space information.
  • The inode list contains the inode table and the kernel references the inode list area to get the information about the files stored on the machine.
  • The data block is the end of the inode list and starting of the blocks that can be used to store the user files. The starting area of the data block will contain some administrative files and information and the later block contains the actual files.

Introduction to system concepts – Process subsystem

A process on unix can be created by executing the fork system call. Only the process 0 is created without using this system call and all other processes are created using the fork system call. (Process 0 was created manually by booting the system.)
The process that executes fork is called the parent process and the process that was created is called the child processes. A process can have many child but only one parent.
Kernel  identifies every process by a unique identifier called the process ID or PID.

Process regions:

Text : The information about the instructions of the process
Data: The uninitialized data members (buffer)
Stack: logical stack frames created during function calls. This is created automatically and grows dynamically.
Since the processes in unix executes in two modes, kernel and user. There are separate stacks for both the modes.
All the processes in the system are identified by PID which are stored in the process table. Every process has an entry in the kernel process table. Every process is allocated the u-area(user area in the main memory. The region is the contiguous area of process addresses.
The processes table entry and u area controls the status information about the process. U area is extension of process table entry.

Context of a process

Context of a process is state. When a process is executing the process has a context of execution. When the process shifts from the running to waiting state the context switching takes place.

Process states

Process state in unix could be one of the following
  • Ready
  • Running in user mode
  • Running in kernel mode
  • Sleeping/waiting
  • Terminated

State transition

state transition

Kernel data structures

The kernel data structures occupy fix size tables rather than dynamically allocated space. This approach has one advantage, the kernel code is simpler but there is one disadvantage of this approach too. It limits the number of entries in these data structures.
So if there are free entries in the kernel data structures we are wasting the potential resources of kernel and if, on the other hand the kernel data structure table is free we need to find a way to notify the processes that something has gone wrong.
The simplicity of kernel code, which is because of this limited size of data structures has far too many advantages than disadvantages.

System Administration

Processes that do various functions performed for the general welfare of the user community. Conceptually there is no difference between administrative process and user process. They use same set of system calls user processes do.
They only differ from user processes in rights and privileges they have. So to summarize, kernel cannot distinguish between kernel process and user process it is just the permission of processes and files that some behave as administrative processes and some behave as the user processes.

Friday, December 12, 2014

The Buffer Cache (unix)

Now we know that the files are stored on the hard drive and the processes can access these files and create new files on the disk. When a process requests for a file the kernel brings the file into the main memory where user process can change, read or access the file. The kernel read the super block to figure out the details about the hard drive and the inode table to find the meta data information of any file. So the kernel reads the inode into the memory whenever any process want to access the data and write it back onto the hard disk when the process is done using the file.

The kernel could read and write the file directly from the hard disk and put it in memory and vice versa but the response time and throughput will be very low in this case because of disks sow data transfer speed.

To minimize the frequency of disk usage/access the kernel keeps a buffer to store the recently accessed files and/or frequently accessed files. This buffer is called the buffer cache.

When the process want to read a file the kernel attempts to read this file in the buffer cache, if the data is found in the buffer cache the data/file is sent to the process. If the file is not found in the buffer cache then the file is read from the disk and then kept in the buffer cache so that it can be made available to the process.

To minimize the disk access frequency the kernel may also implement the pre-caching or write delay functionalities.

Buffer Headers

When the system initializes the kernel allocates the space for the buffer cache. The buffer cache contains two regions/arts. One for the data/files that will be read from the disk, second the buffer header.

The data in the buffer cache corresponds to the logical blocks of the disk block of file system. The buffer cache is “in memory” representation of the disk blocks. This mapping is temporary as the kernel may wish t load some other files’ data into the cache at some later stage.

There will never be a case when the buffer has two entries for the same file on disk as this could lead to inconsistencies. There is only and only one copy of a file in the buffer.

The buffer header contains the metadata information like device number and the block number range for which this buffer holds the data. It stores the logical device number and not the physical device number. The buffer header also contains pointer to a data array for the buffer (i.e. pointer to the data region) .
The buffer header also contains the status of the buffer. The status of the buffer could be
  • Locked/unlocked
  • Buffer contains a valid data or not.
  • Whether the kernel should write the contents to disk immediately or before reassigning the buffer(write delay)
  • Kernel is currently reading the data or writing the data.
  • Is there any process waiting for the buffer to get free.
unix img

Structure of the buffer pool

The kernel caches the least recently used data into the buffer pool. Once a balck from buffer pool is allocated for a file of the system this bliock cannot be used for any other file’s data. The kernel also maintains a free list of buffers. The free list is a doubly circular list of buffers.

When kernel wants to allocate any buffer it removes a node from the free list, usually from the beginning of list but is could take it from middle of the list too. When kernel frees a node from the buffer list it adds this free node at the end of the free list.
unix img
When kernel want to access the disk it searches the buffer pool for a particular device number-block number combination (which is maintained in the buffer header).  The entire buffer pool is organized as queues hashed as a function of device number-block number combination. The figure down below shows the buffers on their hash queues.
unix img
The important thing to note here is that no two nodes in the buffer pool can contain the data of same disk block i.e. same file.

Scenarios of retrieval of buffer

High level kernel algorithms in file subsystem invoke the algorithms of buffer pool to manage the buffer cache.  The algorithm for reading and writing disk blocks uses the algorithm getblk to allocate buffer from the pool.
unix img
The five typical scenarios that kernel may follow in getblk to allocate a buffer in the disk block are
  • Block in the hash queue, and its buffer is free.
  • Cannot find block on the hash queue => allocate a buffer from free list.
  • Cannot find block on the hash queue => allocate a buffer from free list but buffer on the free list marked “delayed write” => flush “delayed write” buffer and allocate another buffer.
  • Cannot find block on the hash queue and free list of buffer also empty.
  • Block in the hash queue, but buffer is busy.
The first scenario - Block in the hash queue, and its buffer is free.
unix img
unix img
Before continuing to other scenarios lets see what happens after the buffer is allocated. The kernel may read the data, manipulate it and/or change it in the buffer. While doing so the kernel marks the buffer as busy so that no other process can access this block. When the kernel is done using this block it releases the buffer using brelse algorithm.
unix img
The second scenario - Cannot find block on the hash queue => allocate a buffer from free list.
unix img
unix img
The third scenario - Cannot find block on the hash queue => allocate a buffer from free list but buffer on the free list marked “delayed write” => flush “delayed write” buffer and allocate another buffer.
unix img
unix img
The fourth scenario - Cannot find block on the hash queue and free list of buffer also empty.
unix img
  • The fifth scenario - Block in the hash queue, but buffer is busy.
unix img

Algorithms for Reading and writing disk blocks

unix img
unix img
unix img

Advantages of the buffer cache

  • Uniform disk access => system design simpler
  • Copying data from user buffers to system buffers => eliminates the need for special alignment of user buffers.
  • Use of the buffer cache can reduce the amount of disk traffic.
  • Single image of of disk blocks contained in the cache => helps insure file system integrity

Disadvantages of the buffer cache

  • Delayed write => vulnerable to crashes that leave disk data in incorrect state
  • An extra data copy when reading and writing to and from user processes => slow down when transmitting large data

Tuesday, December 9, 2014

Installation of Redhat Jboss EAP 6.2 (Enterprise Application Platform)

After Oracle weblogic and Websphere application server I got an opportunity to work with jboss application server. As it is very basic application server its installation is much easier than that of weblogic and websphere.  Before moving further lets have glance on JBOSS EAP and JBOSS Community.  

Jboss EAP (Enterprise Application Platform) and Jboss Community (Open Source) are the same process of installation, configuration, and management. The main difference is that the EAP one is a License version and you required the valid Redhat Support credential to download it and it comes with Red Hat JBoss patches and bug Fixes. EAP is more stable than that of Jboss Community version and you can raise a call to Redhat in case of any issue.

Here is the scenario where we are going to install the JBoss EAP 6.2 in our environment and configure and we will follow the procedure given bellow. 
  1. Check system prerequisites 
  2. Install JBoss EAP 6.2 using zip download  
  3. Add user for Management
  4.  Start / Stop JBOSS EAP 6.2 as a Standalone Server
  5. Access and Manage JBoss EAP 6.2 using the Management Console and Management CLI

Step 1 : Check System Prerequisites

Before proceeding with the installation, you should have a installable available with you

i.e. jboss-eap-6.2.0.zip

Next is to make sure that you have a JVM installed in your machine. You can try running the following command in your terminal:

$ java –version


In case if you don’t have a JVM install, the install suitable JVM first.

Step 2 : Install JBoss EAP 6.2 using zip download


There are several ways to install JBoss Enterprise Application Platform 6.2. We’ll go with the simplest form which is using the zip download. You may install JBoss EAP 6.2 on any location of your machine.

Download the JBoss Enterprise Application Platform 6.2 from redhat, under Downloads -> JBoss Enterprise Middleware -> Application Platform, at the very bottom. Unzip it to your home directory, this will be our EAP_HOME.



Step 3 : Add user for Management


Before you could use the Management Console and/or Management CLI, we have to setup a management user first.
  • open up terminal and go to EAP_HOME/bin
  • run add-user.bat


  • Once you see the screen above simply hit enter until you are asked to enter the username
  • Enter desired username and password, answer YES on the next questions

Step 4 : Start / Stop JBOSS EAP 6.2 as a Standalone Server


Before Start the jboss server you have to set JAVA_HOME and JBOSS_HOME parameter in Windows environment variable (in case of windows environment) or in the cmd window from where you are running the jboss server.
set JAVA_HOME=Path of jdk installed
set JBOSS_HOME=Path of Jboss Directory


  • open up terminal and go to EAP_HOME/bin
  •  run command $ standalone.bat

  • close terminal to stop server or simply hit ctrl + c

Step 5 : Access and Manage JBoss EAP 6.2 using the Management Console and Management CLI

Management Console

Open a web browser and type http://localhost:9990/


Management CLI
  • open up terminal and go to EAP_HOME/bin
  •  run command $ jboss-cli.bat  --connect

  • run exit to quite


I hope this post will help you all

Saturday, December 6, 2014

PowerShell commands to learn

PowerShell combines the speed of the command line with the flexibility of a scripting language, making it a valuable Windows administration tool. Here are a few basic commands you'll want to master.
Over the last few years, Microsoft has been trying to make PowerShell the management tool of choice. Almost all the newer Microsoft server products require PowerShell, and there are lots of management tasks that can't be accomplished without delving into the command line. As a Windows administrator, you need to be familiar with the basics of using PowerShell. Here are 10 commands to get you started.

1: Get-Help

The first PowerShell cmdlet every administrator should learn is Get-Help. You can use this command to get help with any other command. For example, if you want to know how the Get-Process command works, you can type:
Get-Help -Name Get-Process
and Windows will display the full command syntax.
You can also use Get-Help with individual nouns and verbs. For example, to find out all the commands you can use with the Get verb, type:
Get-Help -Name Get-*

2: Set-ExecutionPolicy

Although you can create and execute PowerShell scripts, Microsoft has disabled scripting by default in an effort to prevent malicious code from executing in a PowerShell environment. You can use the Set-ExecutionPolicy command to control the level of security surrounding PowerShell scripts. Four levels of security are available to you:
  • Restricted -- Restricted is the default execution policy and locks PowerShell down so that commands can be entered only interactively. PowerShell scripts are not allowed to run.
  • All Signed -- If the execution policy is set to All Signed then scripts will be allowed to run, but only if they are signed by a trusted publisher.
  • Remote Signed -- If the execution policy is set to Remote Signed, any PowerShell scripts that have been locally created will be allowed to run. Scripts created remotely are allowed to run only if they are signed by a trusted publisher.
  • Unrestricted -- As the name implies, Unrestricted removes all restrictions from the execution policy.
You can set an execution policy by entering the Set-ExecutionPolicy command followed by the name of the policy. For example, if you wanted to allow scripts to run in an unrestricted manner you could type:
Set-ExecutionPolicy Unrestricted

3: Get-ExecutionPolicy

If you're working on an unfamiliar server, you'll need to know what execution policy is in use before you attempt to run a script. You can find out by using the Get-ExecutionPolicy command.

4: Get-Service

The Get-Service command provides a list of all of the services that are installed on the system. If you are interested in a specific service you can append the -Name switch and the name of the service (wildcards are permitted) When you do, Windows will show you the service's state.

5: ConvertTo-HTML

PowerShell can provide a wealth of information about the system, but sometimes you need to do more than just view the information onscreen. Sometimes, it's helpful to create a report you can send to someone. One way of accomplishing this is by using the ConvertTo-HTML command.
To use this command, simply pipe the output from another command into the ConvertTo-HTML command. You will have to use the -Property switch to control which output properties are included in the HTML file and you will have to provide a filename.
To see how this command might be used, think back to the previous section, where we typed Get-Service to create a list of every service that's installed on the system. Now imagine that you want to create an HTML report that lists the name of each service along with its status (regardless of whether the service is running). To do so, you could use the following command:
Get-Service | ConvertTo-HTML -Property Name, Status > C:\services.htm

6: Export-CSV

Just as you can create an HTML report based on PowerShell data, you can also export data from PowerShell into a CSV file that you can open using Microsoft Excel. The syntax is similar to that of converting a command's output to HTML. At a minimum, you must provide an output filename. For example, to export the list of system services to a CSV file, you could use the following command:
Get-Service | Export-CSV c:\service.csv

7: Select-Object

If you tried using the command above, you know that there were numerous properties included in the CSV file. It's often helpful to narrow things down by including only the properties you are really interested in. This is where the Select-Object command comes into play. The Select-Object command allows you to specify specific properties for inclusion. For example, to create a CSV file containing the name of each system service and its status, you could use the following command:
Get-Service | Select-Object Name, Status | Export-CSV c:\service.csv

8: Get-EventLog

You can actually use PowerShell to parse your computer's event logs. There are several parameters available, but you can try out the command by simply providing the -Log switch followed by the name of the log file. For example, to see the Application log, you could use the following command:
Get-EventLog -Log "Application"
Of course, you would rarely use this command in the real world. You're more likely to use other commands to filter the output and dump it to a CSV or an HTML file.

9: Get-Process

Just as you can use the Get-Service command to display a list of all of the system services, you can use the Get-Process command to display a list of all of the processes that are currently running on the system.

10: Stop-Process

Sometimes, a process will freeze up. When this happens, you can use the Get-Process command to get the name or the process ID for the process that has stopped responding. You can then terminate the process by using the Stop-Process command. You can terminate a process based on its name or on its process ID. For example, you could terminate Notepad by using one of the following commands:
Stop-Process -Name notepad

Stop-Process -ID 2668
Keep in mind that the process ID may change from session to session.

Friday, December 5, 2014

Execute Powershell program


Step 1: Open a Notepad and save that file with name "HelloWorld.PS1"

PowerShell files are stored as extension .PS1

Step 2: Type following commands in notepad and save that file.
$strString = "Hello World"
write-host $strString

Step 3: Open PowerShell
Start button > All Programs > Windows PowerShell 

Step 4: Type Path of "HelloWorld.PS1" to open file.
Example: On my PC i have saved file at following path.
C:\MyPowerShell\HelloWorld.PS1

Step 5: How to run Script successfully.
I have received following error on opening "HelloWorld.PS1" file on PowerShell screen.

PS C:\Users\taman01> c:\MyPowerShell\HelloWorld.ps1
File C:\MyPowerShell\HelloWorld.ps1 cannot be loaded because the execution of scripts is disabled on this system. Please see "get-help about_signing" for more details.
At line:1 char:30
+ c:\MyPowerShell\HelloWorld.ps1 <<<<
PS C:\Users\taman01>
 Cause of Error:
The security settings built into Windows PowerShell include something called the “execution policy;” the execution policy determines how (or if) PowerShell runs scripts. By default, PowerShell’s execution policy is set to Restricted; that means that scripts - including those you write yourself - won’t run. Period.

You can verify the settings for your execution policy by typing the following at the PowerShell command prompt and then pressing ENTER:
PS C:\Users\taman01> Get-ExecutionPolicy
Restricted

Solution
If you don’t like the default execution policy (and you probably won’t) then just go ahead and change it. For example, suppose you want to configure PowerShell to run - without question - any scripts that you write yourself, but to run scripts downloaded from the Internet only if those scripts have been signed by a trusted publisher. In that case, use this command to set your execution policy to RemoteSigned.

Type following command on powershell screen.
PS C:\Users\taman01> Set-ExecutionPolicy unrestricted
Set-ExecutionPolicy : Access to the registry key 'HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PowerShell\1\ShellIds\Microsoft.PowerShell' is denied.
At line:1 char:20
+ Set-ExecutionPolicy  <<<< unrestricted

I ran into one more error, this time i was not allowed to change default execution policy.  To resolve this error, you need to open windows powershell as administrator on your windows vista OS.

Type following command after login as administrator
PS C:\Windows\System32> Set-ExecutionPolicy RemoteSigned
PS C:\Windows\System32>

Now lets close Windows Powershell screen as administrator and open again without run as administrator.

This time HelloWorld.PS1 would execute and display "Hello World" on PowerShell Screen.
PS C:\Users\taman01> c:\MyPowerShell\HelloWorld.PS1
Hello World

Windows PowerShell


Microsoft Windows PowerShell is a new command-line shell and scripting language designed for system administration and automation. 

Windows PowerShell is Microsoft's task automation framework, consisting of a command-line shell and associated scripting language built on top of, and integrated with, the .NET Framework. PowerShell provides full access to COM and WMI, enabling administrators to perform administrative tasks on both local and remote Windows systems.

In PowerShell, administrative tasks are generally performed by cmdlets (pronounced command-lets), specialized .NET classes implementing a particular operation. Sets of cmdlets may be combined together in scripts; executables, which are standalone applications; or by instantiating regular .NET classes (or WMI/COM Objects). These work by accessing data in different data stores, like the filesystem or registry, which are made available to the PowerShell runtime via Windows PowerShell providers.

Windows PowerShell also provides a hosting mechanism with which the Windows PowerShell runtime can be embedded inside other applications. These applications then leverage Windows PowerShell functionality to implement certain operations, including those exposed via the graphical interface. This capability has been utilized by Microsoft Exchange Server 2007 to expose its management functionality as PowerShell cmdlets and providers and implement the graphical management tools as PowerShell hosts which invoke the necessary cmdlets. Other Microsoft applications including Microsoft SQL Server 2008 also expose their management interface via PowerShell cmdlets. With PowerShell, graphical interface-based management applications on Windows are layered on top of Windows PowerShell. In the future all Microsoft applications running on the Windows platform are to be PowerShell aware.


Thursday, December 4, 2014

Download Gmail attachments automatically to Google Drive

If you haven’t used Send to Google Drive before, here’s how the it works.
You specify a Gmail label name in the sheet and the script will scan all your existing and incoming email messages inside that label for file attachments. For instance, you can say “inbox” to monitor the inbox folder, specify a custom label name or say “all” to monitor every folder in your Gmail mailbox. If the script finds an attachment, the file will be downloaded to your Google Drive.
Here’s a step by step guide on how you can setup the Send to Google Drive program for your Gmail account. It takes less than a minute.
  1. Click here to create a copy of the Send to Google Drive sheet in your Google Drive.
  2. Open the sheet and you should see a new Gmail Attachments menu at the top. Choose Authorize and grant the necessary permissions. This is an open-source Google Script) that runs in your own Google account and no one else will have access to any of your data.
  3. Click the Gmail Attachments menu again and select Run (step #2). Close the Google sheet and the script will begin downloading attachments from Gmail to your Google Drive in the background.
The program will monitor the specified Gmail label every five minutes and as soon as it finds a message that has attachments, it will automatically save the file(s) to the specified folder in your Google Drive.
Once the attachments have been saved, a new label — Processed — is applied to the Gmail message indicating that the message has been processed by the script. It saves all types of attachments but you can also specify a list of extensions separated by commas.
If you would like to stop the script later, open the same Google Sheet and choose Uninstall from the Gmail Attachments menu.

Source: Labnol.org

Useful Website list

  1. ctrlq.org/screenshots – for capturing screenshots of web pages on mobile and desktops.
  2. dictation.io – online voice recognition in the browser itself.
  3. zerodollarmovies.com – find full-length movies on YouTube.
  4. screenr.com – record movies of your desktop and send them straight to YouTube.
  5. goo.gl – shorten long URLs and convert URLs into QR codes.
  6. unfurlr.come – find the original URL that’s hiding behind a short URL.
  7. qClock – find the local time of a city using Google Maps.
  8. copypastecharacter.com – copy special characters that aren’t on your keyboard.
  9. codeacademy.com – the best place to learn coding online.
  10. lovelycharts.com – create flowcharts, network diagrams, sitemaps, etc.
  11. iconfinder.com – find icons of all sizes.
  12. office.com – download templates, clipart and images for your Office documents.
  13. followupthen.com – the easiest way to setup email reminders.
  14. jotti.org – scan any suspicious file or email attachment for viruses.
  15. wolframalpha.com – gets answers directly without searching   – see more wolfram tips.
  16. printwhatyoulike.com – print web pages without the clutter.
  17. ctrlq.save – save online files to Dropbox or Google Drive directly.
  18. ctrql.rss – a search engine for RSS feeds.
  19. e.ggtimer.com – a simple online timer for your daily needs.
  20. coralcdn.org – if a site is down due to heavy traffic, try accessing it through coral CDN.
  21. random.org – pick random numbers, flip coins, and more.
  22. pdfescape.com – lets you can quickly edit PDFs in the browser itself.
  23. tubemogul.com – simultaneously upload videos to YouTube and other video sites.
  24. scr.im – share you email address online without worrying about spam.
  25. spypig.com – now get read receipts for your email.
  26. myfonts.com/WhatTheFont – quickly determine the font name from an image.
  27. google.com/webfonts – a good collection of open source fonts.
  28. regex.info – find data hidden in your photographs – see more EXIF tools.
  29. livestream.com – broadcast events live over the web, including your desktop screen.
  30. iwantmyname.com – helps you search domains across all TLDs.
  31. homestyler.com – design from scratch or re-model your home in 3d.
  32. join.me – share you screen with anyone over the web.
  33. onlineocr.net – recognize text from scanned PDFs – see other OCR tools.
  34. flightstats.com – Track flight status at airports worldwide.
  35. wetransfer.com – for sharing really big files online.
  36. hundredzeros.com – the site lets you download free Kindle books.
  37. polishmywriting.com – check your writing for spelling or grammatical errors.
  38. marker.to – easily highlight the important parts of a web page for sharing.
  39. typewith.me – work on the same document with multiple people.
  40. whichdateworks.com – planning an event? find a date that works for all.
  41. everytimezone.com – a less confusing view of the world time zones.
  42. gtmetrix.com – the perfect tool for measuring your site performance online.
  43. noteflight.com – print music sheets, write your own music online (review).
  44. imo.im – chat with your buddies on Skype, Facebook, Google Talk, etc. from one place.
  45. translate.google.com – translate web pages, PDFs and Office documents.
  46. kleki.com – create paintings and sketches with a wide variety of brushes.
  47. similarsites.com – discover new sites that are similar to what you like already.
  48. wordle.net – quick summarize long pieces of text with tag clouds.
  49. bubbl.us – create mind-maps, brainstorm ideas in the browser.
  50. kuler.adobe.com – get color ideas, also extract colors from photographs.
  51. liveshare.com – share your photos in an album instantly.
  52. lmgtfy.com – when your friends are too lazy to use Google on their own.
  53. midomi.com – when you need to find the name of a song.
  54. google.com/history – see your past searches, also among most important Google URLs
  55. bing.com/images – automatically find perfectly-sized wallpapers for mobiles.
  56. faxzero.com – send an online fax for free – see more fax services.
  57. feedmyinbox.com – get RSS feeds as an email newsletter.
  58. ge.tt – qiuckly send a file to someone, they can even preview it before downloading.
  59. pipebytes.com – transfer files of any size without uploading to a third-party server.
  60. tinychat.com – setup a private chat room in micro-seconds.
  61. privnote.com – create text notes that will self-destruct after being read.
  62. boxoh.com – track the status of any shipment on Google Maps – alternative.
  63. mondrian.io – create vector drawings in the browser
  64. draw.io – create diagrams and flowcharts in the browser, export your drawings to Google Drive and Dropbox.
  65. downforeveryoneorjustme.com – find if your favorite website is offline or not?
  66. ewhois.com – find the other websites of a person with reverse Analytics lookup.
  67. whoishostingthis.com – find the web host of any website.
  68. labnol.org – software tutorials and how-to guides.
  69. disposablewebpage.com – create a temporary web page that self-destruct.
  70. urbandictionary.com – find definitions of slangs and informal words.
  71. seatguru.com – consult this site before choosing a seat for your next flight.
  72. unsplash.comdownload images absolutely free.
  73. zoom.it – view very high-resolution images in your browser without scrolling.
  74. scribblemaps.com – create custom Google Maps easily.
  75. alertful.com – quickly setup email reminders for important events.
  76. picmonkey.com – Picnik is offline but PicMonkey is an even better image editor.
  77. formspring.me – you can ask or answer personal questions here.
  78. sumopaint.com – an excellent layer-based online image editor.
  79. snopes.com – find if that email offer you received is real or just another scam.
  80. typingweb.com – master touch-typing with these practice sessions.
  81. mailvu.com – send video emails to anyone using your web cam.
  82. timerime.com – create timelines with audio, video and images.
  83. stupeflix.com – make a movie out of your images, audio and video clips.
  84. safeweb.norton.com – check the trust level of any website.
  85. teuxdeux.com – a beautiful to-do app that looks like your paper dairy.
  86. deadurl.com – you’ll need this when your bookmarked web pages are deleted.
  87. minutes.io – quickly capture effective notes during meetings.
  88. youtube.com/leanback – Watch YouTube channels in TV mode.
  89. youtube.com/disco – quickly create a video playlist of your favorite artist.
  90. talltweets.com – Send tweets longer than 140 characters.
  91. pancake.io – create a free and simple website using your Dropbox account.
  92. builtwith.com – find the technology stack to know everything about a website.
  93. woorank.com – research a website from the SEO perspective.
  94. mixlr.com – broadcast live audio over the web.
  95. radbox.me – bookmark online videos and watch them later (review).
  96. tagmydoc.com – add QR codes to your documents and presentations (review).
  97. notes.io – the easiest way to write short text notes in the browser.
  98. ctrlq.org/html-mail – send rich-text mails to anyone, anonymously.
  99. fiverr.com – hire people to do little things for $5.
  100. otixo.com – easily manage your online files on Dropbox, Google Docs, etc.
  101. ifttt.com – create a connection between all your online accounts.

Changelog and Updates

The following websites were part of the original list but they are either no longer available or have been replaced with better alternatives.
  1. virustotal.com – scan any suspicious file or email attachment for viruses.
  2. isnsfw.com – when you wish to share a NSFW page but with a warning.
  3. truveo.com – the best place for searching web videos.
  4. tabbloid.com – your favorite blogs delivered as PDFs.
  5. warrick.cs.odu.edu – you’ll need this when your bookmarked web pages are deleted.
  6. tempalias.com – generate temporary email aliases, better than disposable email.
  7. whisperbot.com – send an email without using your own account.
  8. errorlevelanalysis.com – find whether a photo is real or a photoshopped one.
  9. google.com/dictionary – get word meanings, pronunciations and usage examples.
  10. wobzip.org – unzip your compressed files online.
  11. namemytune.com – when you need to find the name of a song.
  12. snapask.com – use email on your phone to find sports scores, read Wikipedia, etc.
  13. pastebin.com – the site has been blocked in India.
  14. encrypted.google.com – Google now redirects all logged-in users to the https version of google.com by default so this is no longer necessary.
  15. bounceapp.com – replace this with a version that works on mobile.
  16. dabbleboard.com – a online virtual whiteboard that will shut down in August 2012.
  17. chipin.com – helps you raise funds online for an event or a cause (closed).


Source: Labnol.org