LINUX VS SOLARIS Performance

Table : Some Libraries in Linux and Solaris OS

Solaris OS	Linux	Description
libc	libc	The standard C library (POSIX, SysV, ANSI, etc.) See man libc on Solaris OS.
libucb	libc	UCB (University California Berkeley) compatibility library
libmalloc	libc	There are several different malloc libraries; the default is in libc.
libsocket	libc	Socket library (sockets are in libc on Linux).
libxnet	libc	X/Open Networking library
libresolv	libresolv	DNS routines (and on Solaris OS, inet_* routines)
libnsl	libnsl/libc	Network services library (linux - nis/nis+ routines)
librpc	librpc	RPC functions
libslp	libslp	Service Location Protocol
libsasl	libsasl	Simple Authentication and Security Layer
libaio	libaio	Asynchronous I/O library
libdoor		Door support ( door_create(), door_return(), etc.)
librt	librt	POSIX Real Time library
libcfgadm		Configuration administration library
libcontract		Contract management library (see man contract.4 on Solaris OS)
libcpc		CPU performance counter library (on Linux, may need to install kernel module?)
libdat		Direct Access Transport Library (see http://www.datcollaborative.org)
libelf	libelf	ELF support library
libm	libm	Math library

LINUX VS SOLARIS PERFORMANCE

Linux : orange color

Solaris : blue color

The next sections take a closer look at some of the system calls and libraries. We'll concentrate on what's different between the systems.

One of these is Linux, the other SmartOS. Same hardware:

• One system is 14% slower

• Imagine that's your system – you'd want to know why

• I recently had a customer with a complex performance issue try

a one-liner like this, as a simple test, and with a similar result.

It's an interesting tour of some system differences

systemA$ time perl -e 'for ($i = 0; $i < 100_000_000; $i++) { $s = "SCaLE12x" }'

real 0m18.534s

user 0m18.450s

sys 0m0.018s

systemB$ time perl -e 'for ($i = 0; $i < 100_000_000; $i++) { $s = "SCaLE12x" }'

real 0m16.253s

user 0m16.230s

sys 0m0.010s

Can the kernel make a difference? ... As a reminder:

• The program makes no system calls during the loop

• Yes, for a number of reasons:

• Setting the string involves memory I/O, and the kernel

controls memory placement. Allocating nearby memory in

a NUMA system can significantly improve performance

• The kernel may also control the CPU clock speed (eg, Intel

SpeedStep), and vary it for temp or power reasons

• The program could be perturbed by interrupts: eg, network

I/O (although the performance effect should be small).

During a perturbation, the kernel CPU scheduler may

migrate the thread to another CPU, which can hurt

performance (cold caches, memory locality)

Big Differences: Linux

Up-to-date packages Latest application versions, with the latest

performance fixes

Large community Weird perf issue? May be answered on

stackoverflow, or discussed at meetups

More device drivers There can be better coverage for high

performing network cards or driver features

futex Fast user-space mutex

RCU Fast-performing read-copy updates

btrfs Modern file system with pooled storage

DynTicks Dynamic ticks: tickless kernel, reduces

interrupts and saves power

SLUB Simplified version of SLAB kernel memory

allocator, improving performance

System Similarities

• It's important to note that many performance-related features

are roughly equivalent:

• Both are Unix-like systems: processes, kernel, syscalls,

time sharing, preemption, virtual memory, paged virtual

memory, demand paging, ...

• Similar modern features: unified buffer cache, memory

mapped files, multiprocessor support, CPU scheduling

classes, CPU sets, 64-bit support, memory locality,

resource controls, PIC profiler, epoll, ...

What Solaris can learn from Linux performance?

• Packaging

• Community

• Compiler Options

• likely()/unlikely()

• Tickless Kernel

• Process Swapping

• Either learning what to do, or learning what not to do...

• Overcommit & OOM Killer

• SLUB

• Lazy TLB

• TIME_WAIT Recycling

• sar

• KVM

What Linux can learn from Solaris performance?

• ZFS

• Zones

• STREAMS

• Symbols

• prstat -mLc

• vfsstat

• DTrace

• Culture

Lead Performance Engineer

brendan@joyent.com

Brendan Gregg

@brendangregg

Linux Runlevels

Runlevels define what tasks can be accomplished in the current state (or runlevel) of a Linux system. Every Linux system supports three basic runlevels, plus one or more runlevels for normal operation. The basic runlevels are shown in Table 1.

Table 1. Linux basic runlevels

Level	Purpose
0	Shut down (or halt) the system
1	Single-user mode; usually aliased as s or S
6	Reboot the system

Beyond the basics, runlevel usage differs among distributions. One common usage set is shown in Table 2.

Table 2. Other common Linux runlevels

Level	Purpose
2	Multiuser mode without networking
3	Multiuser mode with networking
5	Multiuser mode with networking and the X Window System

Linux Services

List running services:

# service --status-all
# chkconfig --list | grep '3:on'

Recommend Actions on Default Services

Service	Description	Action
acpid	Advanced Configuration and Power Interface event daemon	Enable
anacron	Anacron is like cron, but it does not assume that the machine is running continuously. Hence, it can be used on machines that aren’t running 24 hours a day, to control daily, weekly, and monthly jobs that are usually controlled by cron.	Disable on servers
apmd	Advanced Power Management Subsystem (old system). If the server is capable of ACPI support, disable this service	Disable if possible
auditd	The Linux Auditing System	Enable and configure
atd	atd runs jobs queued by at	Enable and configure
autofs	The automount(8) program is used to manage mount points for autofs, the inlined Linux automounter. You can mount NFS, USB, DVD/CD, and CIFS via /etc/fstab.	Disable
avahi-daemon and avahi-dnsconfd	The Avahi mDNS/DNS-SD daemon implementing Apple's ZeroConf architecture (also known as "Rendezvous" or "Bonjour").	Disable
bluetooth and hidd	Bluetooth services for service discovery, authentication, Human Interface Devices (hidd), etc	Disable
cpuspeed	This service monitors the system’s idle percentage and reduces or raises the CPUs’ clock speeds and voltages accordingly to minimize power consumption when idle and maximize performance when needed	Enable
crond	Service to execute scheduled commands via crond daemon.	Enable and configure
cups	Common unix printing system service	Disable if possible
dc_client & dc_client	Startup script for the Distcache SSL Session Cache Client and server proxy. Disable if you do not need cache proxy.	Disable if possible
dnsmasq	the DNS caching server. Enable if your ISP or remote DNS caching server is pretty slow.	Enable
dkmdkms_autoinstallers	dkms is a framework which allows kernel modules to be dynamically built for each kernel on your system in a simplified and organized fashion.	Disable if possible
firstboot	RHLE specific service. It does a few configuration following successful installation of the operating system.	Disable
gpm	A cut and paste utility and mouse server service for virtual consoles.	Disable
haldaemon	This service is used for collecting and maintaing information about hardware from several sources. This is only used for X and desktop apps. Disable it on servers.	Disable
hplip	A service for non-PostScript HP printer. Disable it on servers.	Disable
irda	IrDA(TM) (Infrared Data Association) is an industry standard for wireless, infrared communication between devices. IrDA speeds range from 9600 bps to 4 Mbps, and IrDA can be used by many modern devices including laptops, LAN adapters, PDAs, printers, and mobile phones.	Disable if possible
iscsi & iscsid	iscsi service logs into iSCSI targets needed at system startup (i.e. iscsi client). iscsid will start and stop iSCSI daemon. Use this if you've iscsi based storage.	Disable if possible
iptables & ip6tables	IPv4 and IPv6 firewall service.	Enableandconfigure
irqbalance	The irqbalance service will distribute interrupts across the cpus on a multiprocessor system with the purpose of spreading the load.	Enable
isdn	Provides the Internet connectivity using an ISDN modem.	Disable if not using an ISDN modem.
kdump	Kernel crash dump analyzer. This service is useful for kernel hackers and device driver development or testing new kernel feature. Dsable the service on production boxes.	Disable
kudzu	RHEL specific hardware detection service. This is required on desktop or laptop where end users can add a new hardware but not on servers.	Disable
lm_sensors	lm_sensors is used for monitoring motherboard sensor values.	Disable
lvm2-monitor	Starts and stops dmeventd monitoring for lvm2. If you are not using LVM2 (Linux volume manager) based storage disable it.	Disable
mcstrans	Starts the SELinux Context Translation System daemon. This is site specific SELinux requirements.	Disable if possible
mdmonitor	software RAID monitoring and management service. If you are not using software RAID disable it. This is not required for hardware RAID setup as they comes with their own programs.	Disable if possible
messagebus	This service broadcasts notifications of system events and other messages (D-bus). Turn it on for bluetooth, X Windows and desktop systems.	Disable
microcode ctl	Script to apply cpu microcode for Intel IA32 processor. If you are not using Intel IA32 processor disable it.	Disable
netfs, nfslock, rpcgssd, rpcidmapd, and portmap	Mount and configure Linux network network filesystems (NFS). If you are not using NFS client/server technology disable it.	Disable if possible
network	A service to activates/deactivates all network interfaces configured to start at boot time.	Enable
pcscd	The PC/SC smart card sevice is a resource manager for the PC/SC lite and Musclecard frameworks. It coordinatescommunications with smart card readers, smart cards, and cryptographic tokens that are connected to the system.If Smart Cards are not in use on the system, disable this service:	Disable
readahead_early and readahead_later	This sevice provide one-time caching of files belonging to a few boot services. It does not provide any boosting. Just disable it.	Disable
restorecond	This service restores the correct security context for SELinux.	Enable
rhnsd	This service handles the task of connecting periodically to the RHN servers to check for updates, notifications and perform system monitoring tasks according to the service level that your server is subscribed for. Disable this service and use yum-updatesd service.	Disable
sendmail	Use to start sendmail server.	Enable and configure
smartd	Self Monitoring and Reporting Technology (SMART) Daemon for hard disks.	Enable and configure
setroubleshoot	This service starts the SELinux Troubleshooting daemon. It will send notification tothe desktop user of SELinux access denied messages in a user-friendly fashion.	Disable
sshd	The openssh server. If you need remote login enable it. You must enable this on all servers so that you can login and configure everything.	Enable
syslog	Syslog is the facility by which many other Linux daemons use to log messages to various system log files. It is a good idea to always run syslog.	Enable
xfs	X Windows font server. Disable it on servers.	Disable
yum-updatesd	Update notification daemon for system packages.	Enable

Remove Outdated Insecure Services

Is there a mission-critical reason for users to access the system via the insecure protocols such as ftp, NIS and telnet? The following services are obsolete services and must be deleted for security:

1. inetd and xinetd : This is not installed by default. But, it it is installed just delete it. Consider switching to more secure services which provide the needed functionality.
2. telnet-server : Delete insecure telnet remote login, use OpenSSH server and ssh client to get back into the server.
3. rsh-server : Delete insecure rlogin, rsh, or rcp commands. Use scp and ssh commands from the OpenSSH.
4. ypserv & ypbind : Remote outdated NIS, consider using OpenLDAP or Fedora / Redhat directory server.
5. tftp-server : Remove outdated and insecure TFTP server software.

Configure Required Services

Other services need to be installed and configr as and when required:

1. httpd: The Apache web server.
2. php-cgi: The php server.
3. bind9 (named): The DNS server.
4. ntpd : The time network based time client/server.
5. snmpd : The net-snmp server.
6. squid : The squid proxy and web caching server.

Monitoring tools in Linux

Features of network monitoring tools
A network monitoring tool is usually hosted on a standalone server and runs its client software on each machine to be managed or monitored. The tool usually runs its own copy of the database such as MySQL or Postgres, which stores all scripts, historic events and actions. In some modern tools, an agent is not required to be run on managed machines, making it an agent-less installation. Table 1 lists a stack of features with examples, which must be available by default in a network monitoring tool.

When it comes to monitoring large scale IT infrastructures, systems administrators need architecture with much-advanced features to make their life easy. Given below is a list of some important features.

Auto discovery: It is cumbersome for administrators to add each managed device manually. Modern monitoring tools span the entire network segment to enumerate devices and perform auto discovery of operating system, configuration and settings. This feature automatically helps admins to get a glimpse of their IT inventory.
Network traffic stats: Earlier, monitoring tools used to just look at the CPU, memory and disk utilisation. However, this is not enough and network bandwidth usage is a key factor to be aware of, especially when the managed machines are supposed to access the Internet. Besides, by monitoring network traffic, admins get an insight into the bandwidth usage of the Internet service provider’s line, which helps them make a capacity planning decision.
Log monitoring: All operating systems create activity logs. For example, in case of Linux, SSH logs and bash logs are created, while for Windows, the application, system and security event logs are generated. A good tool must be capable of reading and parsing log files. This sounds easy but can be tricky, because the operating system opens log files and locks those that require tools to sneak into the file without tampering or corrupting it. Monitoring tools should be able to check log file size, parse text for particular string patterns, etc, and perform configured actions. This gives a lot of power to admins to tune their infrastructure monitoring for better control.
Device grouping: This is important for easy management of devices such as firewalls, servers, etc, in specific groups. In some cases, administrators choose to create department wise groups, or a group for each building or floor. They populate these groups with network switches, servers and desktops pertaining to that department or floor. In a growing infrastructure, this feature is very important.
Alert management: Merely monitoring a network is not enough. A good tool should let admins produce alerts. For example, if the CPU of a critical server crosses 90 per cent of usage, or if a firewall is dropping multiple packets in a row, it should create a trouble ticket, and email, or optionally send a short message to the admin’s mobile phone. Almost all tools provide such facilities today to enhance their usefulness; however, admins should look into the configurability and facilities available in alert management, prior to selecting the proper tool.
Customisable Web dashboard: A good monitoring tool should let admins access its statistics over a Web interface. Besides, the Web interface must be customisable to let them decide what should be on the dashboard’s front page. Modern tools provide widgets that are small screen sections or windows, which can show monitoring statistics of the admin’s choice and can be moved or removed.
Integrating with helpdesk: Recording of events that are the result of threshold violation is very important and should be an automated process. The monitoring tool should provide the necessary hooks or connectors so that the trouble ticket/helpdesk system can be easily connected. Monitoring of events and the applicable actions should result in a trouble ticket. This helps decide how much manpower ought to be utilised to address those events and intelligent action can be taken based on that data.
Report generation: All monitoring tools today provide some level of report generation, which is based on the date, time, etc. However, a detailed report that is device-specific or event-specific is really essential for an admin. For example, a report generator should be able to drill down into a particular event such as a TCP timeout on a particular server, and provide historic occurrences of that event for that server. These levels of granular details help administrators establish a co-relation between the event and its root cause.
The following is a list of new features found in commercial monitoring tools; however, the open source world will surely catch up in the days to come.
Plug-in API support: While a few open source tools do provide this, there is still a scope for improvement. API calls of the monitoring engine can be exposed in a secure way, so that developers can write their own plug-ins. This is especially important when there is a new network device or software application in the market that must be monitored.
Trend analysis: The network or server monitoring industry is rapidly moving away from the preventive to the pro-active mode. Administrators want to know historic trends of problems and make a judgment in terms of the corrective actions to be taken today, to prevent problems that might happen tomorrow. For example, continuously high CPU utilisation on a MySQL server over a period suggests that one or more stored procedures are either not optimised or misbehaving. This can be related to the application that uses those procedures. Thus, if that application is expected to be used more, an analysis of the trends can tell admins that the MySQL server is going to run into trouble.
Security monitoring: Very soon, no monitoring tool will be useful unless it supports cyber security monitoring. Attacks happening at Layer 2 and 3, as well as application-based security problems at Layer 7, should be trapped and reported by a good monitoring tool. This functionality is available in a few commercial tools; however, incorporating Snort along with Nagios or any other monitoring tool can prove to be a powerful security monitoring solution.
Nagios, Zenoss and Zabbix
So let’s talk about the three famous open source monitoring tools-Nagios, Zenoss and Zabbix, and compare them. While there are many features to compare, we will discuss only those that matter the most to mid-scale IT infrastructure management.
Nagios: This is a famous first generation network monitoring tool and is used in all Linux distros. Developed in C and PHP, it supports multiple flavours of open source backend databases, as well as the legacy flat file structure.
Zenoss: Written using Python scripting, Zenoss provides a highly flexible monitoring platform for mid scale and large scale infrastructures. It supersedes Nagios in a few cases, especially when it comes to alert management.
Zabbix: This is really an enterprise class open source tool. Written in C and PHP, it has very elaborate dashboards that provide admins a detailed drill down.

While it is tough to decide which tool is best for monitoring, here are a few guidelines. Administrators should first look at their infrastructure from the uptime perspective and decide what needs to be really monitored, rather than checking all that they can possibly monitor. This focused approach is important because it is easy to get distracted with multiple features available in each tool. Hence, focusing on the basic monitoring requirements mentioned earlier should be first on the agenda. As a second step, admins should look into the applications to be monitored and decide whether or not custom scripting needs to be done to achieve what they need from the monitoring standpoint. The third step should be to focus on reporting and trend analysis, because as infrastructure grows, it is essential to have a historic record of the problems in IT infrastructure.

The last but important step would be to see if security monitoring is a requirement in the given scenario. If yes, then it is crucial to decide the level of additional scripting and log generation that would be required. The generated log can then be captured by a monitoring tool and report as a problem incidence via the trouble ticketing system.

Nagio, Zenoss and Zabbix are all industry grade, professional tools with large installation bases. It has been observed that Nagios and Zenoss perform very well on the Ubuntu platform, while Zabbix runs great on other distros. Zenoss is unique among the three tools compared, because it offers more features, interacts well with multiple databases and other tools, and also has proved itself to be a robust solution even for high performing, large scale IT infrastructures. Besides these three, there are tools such as Cacti, OpenNMS, Cricket, etc, which I leave readers to find more about on the Net. It is always better to compare an open source tool with a commercial one, and decide and choose the required features.