Using rtorrent / the linux command line torrent client

The following is the shortest tutorial that should get you up and running…

1- install rtorrent

apt-get install rtorrent

2- download the torrents

wget http://cdimage.debian.org/debian-cd/7.0.0/amd64/bt-dvd/debian-7.0.0-amd64-DVD-1.iso.torrent

3- start rtorrent

rtorrent

Hit enter on the screen you are on, then hit tab, you should see a list of the torrent files you downloaded, type the first letter of your file then tab again, the program will either show you your file , or reduce the list to all the files that have that same prefix, add one more letter then hit tab again, keep going until you have your file.

Once done, use the arrow keys to select your torrent file, then hit ctrl+s to Start the torrent.

That’s all you need to know !

Disk load and IO wait on linux

Using the top command, you can find out how much time your processor spends waiting for input and output from disk, the IO wait field, but how can we know what applications are responsible for that IO wait.

to examin disk io and iowait more closely, we need some tools, the following is a list of tools that can show you exactly what is causing all the IO wait so that you can start from there.

Sometimes, apps that use the disk with many random reads should go to the secondary disk. but to find that application, take a look at the following tools, i will be explaining each and every one of them as we go.

Here are a bunch of programs on linux that will tell you just that

  • iotop – apt-get install iotop – simple top-like I/O monitor
  • iostat – apt-get install sysstat – shows and monitors activity per disk – The sysstat package contains the sar, mpstat and iostat commands for Linux.
  • isag – apt-get install isag – Interactive System Activity Grapher for sysstat
  • dstat – apt-get install dstat – versatile resource statistics tool
  • vmstat 1
  • iostat 1
  • lsof
  • strace -e trace=open
  • strace -e trace=open -p
  • In ps auxf you’ll also see which processes are are in uninterruptible disk sleep (D) because they are waiting for I/O.
  • fatrace – Amazing tool showing you which files are being accessed or written on the hard drive

examples

iostat -c -d -x -t 2 100

Gigabit Ethernet over USB 3 with a hub

I got myself a new C850B797 toshiba laptop with an i7-3630QM processor and 16GB of ram (coz it’s cheap), the C850B797 is the grey edition in the C850 series, the processor is a 22nm Lithography and everything is very good as far as i can see, but then i realized that the laptop had one very serious issue, i am limited to 100mbit Ethernet, and i really do move plenty of files around all the time (many gigabytes), so i needed to get gigabit, but how ?

Usb speed is 480mbit/second, much of that goes overhead, so the Ethernet controller would probably be USB3 which would block my only usb port, so the answer is to get a USB3 hub and connect the USB gigabit ethernet to it, but it turns out that the hub followed by gigabit eithernet is already on the market from one of 2 providers.

The first, the one i got is the StarTech ST3300U3S USB hub and Ethernet controller, that uses ASIX AX88179 for gigabit Ethernet after a VIA VL811 USB3 hub, the question is, will i be getting the speeds i am hoping for from this ?

I will be running tests concerning reliability of the combo (since both are connected in serial) and let you guys know.

1- StarTech ST3300U3S USB hub and Ethernet controller – Power adapter included = $55, Also happens to be the same exact thing as the SYBA inf@zone (infozone) with the same chips and the same plastic casing, the SYBA infozone will sell at $33 if you buy 3 together, i found 3 for $98.99 on ebay, making it much cheaper than the $66 each for the same exact thing from StarTech, SYBA labels it as SY-HUB50056.)

AX88179 — USB3.0 to 10/100/1000M Gigabit Ethernet Controller
VIA VL811 – SuperSpeed USB 3.0 Hub Controller

The other options on the market are…
2- Kanex DualRole Gigabit Ethernet with 3-Port USB Hub (USB3GBITX) – Power adapter not included – $40
AX88179 — USB3.0 to 10/100/1000M Gigabit Ethernet Controller
No idea about the hub chip yet.

Please bare in mind that this same combo but with USB 2 and a 100Mbit eithernet adapter should not cost more than $10, it is the USB 3 and gigabit Ethernet that are expected to be this expensive.

Via also produces the VL812 which is almost identical to VL811

ZoneMinder on debian with a DVR card using techwell chipset

1- Installed the debian base system, debian wheezy (7) (But should work with debian squeeze (6))
2- apt-get install ssh openssh-server fail2ban
3- INSTALLED THE HARDWARE 8 CAMERA DVR CARD into the PCI slot
4- Modified the network interfaces to use a static IP, and pointed the DMZ of the router to that IP
5- installed some software (Even though some of them would have been installed anyway when i install zoneminder), but i do it anyway just in case they are optional in zoneminder

apt-get update
apt-get install apache2 php5-mysql libapache2-mod-php5 mysql-server
apt-get install ffmpeg libarchive-tar-perl libarchive-zip-perl libdate-manip-perl libdevice-serialport-perl
apt-get install libjpeg62 libmime-perl libstdc++6 libwww-perl zlib1g
apt-get install zoneminder

I can see close to the end of operation

ZoneMinder is stopped
invoke-rc.d: initscript zoneminder, action “status” failed.
Starting ZoneMinder: success

I hope it’s nothing too serious

Now, check if your card is supported, you issue lspci -v as root… Since my 8 port card is supported, i see eight stanzas of this

I got eight of the following labeled 03:02.0 03:02.1 03:02.2 03:02.3 03:02.4 03:02.5 03:02.6 03:02.7

03:02.2 Multimedia video controller: Techwell Inc. TW6816 multimedia video controller (rev 10)
Subsystem: Techwell Inc. TW6816 multimedia video controller
Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 3
Memory at febfe800 (32-bit, non-prefetchable) [size=1K]
Capabilities: [44] Power Management version 2

If you dont see your card’s devices, well, your out of luck, maybe you should try a newer kernel, i am using 3.2 from debian wheezy (7)

Now, to the second test

ls -ls /dev/video*

NO love, nothing, If i were lucky, i would get something like

ls -ls /dev/video*
0 crw-rw----+ 1 root video 81,  0 30. Jul 15:18 /dev/video0
0 crw-rw----+ 1 root video 81,  2 30. Jul 15:18 /dev/video1
0 crw-rw----+ 1 root video 81,  4 30. Jul 15:43 /dev/video2
0 crw-rw----+ 1 root video 81,  6 30. Jul 15:43 /dev/video3
0 crw-rw----+ 1 root video 81,  8 30. Jul 15:17 /dev/video4
0 crw-rw----+ 1 root video 81, 10 30. Jul 15:17 /dev/video5
0 crw-rw----+ 1 root video 81, 12 30. Jul 15:17 /dev/video6
0 crw-rw----+ 1 root video 81, 14 30. Jul 15:17 /dev/video7

But i did not, so i need the driver to my techwell card.

Simple, there is a nice person who made a driver already, let’s see how we can install it

First, we need GIT so that we can download the driver

apt-get install git
apt-get install linux-headers-$(uname -r)
apt-get install make sudo

(without installing sudo, the compile gives the error /bin/sh: 1: sudo: not found which is because the guy probably wrote the driver with ubuntu in mind)

git clone git://gitorious.org/tw68/tw68-v2.git
cd tw68-v2
make
make install

So, i was expecting 8 cameras, i got the following

ls -ls /dev/video*
0 crw-rw---T 1 root video 81, 0 May 12 23:12 /dev/video0
0 crw-rw---T 1 root video 81, 2 May 12 23:12 /dev/video1
0 crw-rw---T 1 root video 81, 4 May 12 23:12 /dev/video2
0 crw-rw---T 1 root video 81, 6 May 12 23:12 /dev/video3

probably a driver issue, oh well, lets see what we can do with 4 then i will see about this issue once i have 4 working:D

ln -s /etc/zm/apache.conf /etc/apache2/conf.d/zoneminder.conf
adduser www-data video
/etc/init.d/apache2 restart

Now, visiting 192.168.2.101/zm/ allows me to add cameras, the matter is very intuitive from that point on, all you need is a browser, remember you will need to password protect things too :D so that people from the outside can not look at your cameras.

PHP code protection, obfuscation, and encoding

Update:

BCOMPILER is no longer supported past PHP 5.3, BUT, the same developer went back to developing APC, and now, there are APC functions that can do what bcompiler used to do.
I will come back to explain how to use it later (it works perfectly for me without the original files all together. the trick is 2 functions, apc_bin_load and apc_bin_dump.

The other good option is PHC (of phpcompiler.org).

A third option would be obfusc.com

End of update (2014-05-10)
—————————–

For encoding, there are 3 options

Before i waste your time, what i do is this, i obfuscate the code so that the function names, variable names, and other things are useless and meaningless to the reader, then i use bcompiler. Now here are the choices and programs i used through the years.

Zend Guard (formerly Zend Encoder)- Expensive ($600) – Some say they have cracked it but i have never seen any evidence of that, requires the free Zend Optimizer or Zend Guard Loader to run the encoded code
Ioncube – I have a license for this one, does what zend encoder does, needs it’s loader on the server to work, provides a cheaper encode-online edition for those with small projects, lowest license is for 200 dollars
bcompiler – Free, and probably faster than the other two, it is simply native PHP opcode, meaning you spare PHP the need to compile your code (interpret), making your code run faster, but reversing it is not very hard. to make reversing harder, you need to obfuscate your code (change variable names and function/method names to make the code unreadable, read below for software to obfuscate that are available online.
BENCODER v1.6 – Encode your PHP script using bcompiler – to facilitate using bcompiler

PHP Obfuscators

Here are just a few, if you google PHP obfuscator you will find hundreds 😀

truebug – $45
codeeclipse – free service, obfuscation happens online on the website
phpprotect.info – Simple free tool that obfuscates variable names, which is enough most of the time, it is very hard to read code with non descriptive variable names
iddin.sourceforge.net – all in German, but translating reveals that it is an obfuscator

Mounting VMDK files into host when in split file

I usually use OSFmount for mounting files made with DD, ddrescue, dd_rescue and gddrescue, for virtual machines in vmware i can mount those directly with OSFMount as long as it is a single file disk, when the thing has multiple files, i can not do that

Turns out vmware gives a way a tool to consolidate those disks into 1 disk.

cd C:Program Files (x86)VMwareVMware Workstation
vmware-vdiskmanager -r c:my_in_file.vmdk -t 0 file_out.vmdk

Now i can mount them directly onto my host machine.

Note that, even the  http://www.vmware.com/support/developer/vddk/ (The vmware vmdk development kit) which is a recent tool can not mount mutiple split disks, and requieres a single disk to mount on host

Memory in openVZ – all you need to know

First, calculating memory

I want 1GB of ram, how many pages would those be ?
To calculate the numebrs for an openvz conf file, you should know that in container config files, the numbers there are pages, not bytes, you should probably know that a page of memory is 4k which is 4096 bytes

So a gigabyte is 1073741824 bytes

Number of pages = 1073741824 / 4096

So a gigabyte is 262144 pages (which open VZ should get in the config file to end up with a gigabyte of ram).

Now To what values in the openvz config file mean.

So, a memory block in oppenvz is measured by pages not bytes or megabytes, so the word pages generally means memory, now let us look at some variables in the openvz config file, vm means virtual machine (which is your container)

Values are specified in the form of
parameter=”barrier:limit”
For example
VMGUARPAGES=”67584:2147483647″

vmguarpages = VM Guaranteed Pages, the barrier is the effective number (The limit should be set to maximum integer, 2147483647 on 32bit and 9223372036854775807 on 64bit even though it is not defined or useful yet), If the current amount of allocated memory space does not exceed the guaranteed amount (the barrier of vmguarpages), memory allocations of container’s applications always succeed. If the current amount of allocated memory space exceeds the guarantee but below the barrier of privvmpages, allocations may or may not succeed, depending on the total amount of available memory in the system. if allocations exceed privvmpages, allocations will fail.
oomguarpages = Out Of Memory Guaranteed pages (Total for all machines should not exceed the available memory on the server) this is like the above, but assuming the system has no choice but to kill procesees, the system will weigh this in to chose what VM to kill from.
privvmpages = Private VM pages (The barrier and the limit of privvmpages parameter impose an upper limit on the memory allocations)

Eclipse Content assist and fast auto complete

So in my Window => preferences window, i searched for “content assist”, and under content assist for PHP, i made the value 50 rather than 200, this did not help at all as there was no change.

So i searched for text editors and enrich after delay to enrich immediately, and magically everything worked on PDT, this does not make sense, what does hovering have to do with content assist, but there you are, now as soon as i write anything, choices star appearing 😀

Again, Switching from Zend Studio (paid) to PDT (Eclipse) which is ironically made by Zend as well was a very good move, i feel no issues whatsoever, i think this one is even better than the old version of Zend studio i was using, and Zend studio offered me no advantage at all, even though it would probably be beneficial to some who know hot to use the zend tools and get the studio to compliment those tools.

Creating a self signed security certificate

This post is very outdated, you might want to check a more recent version of it at (Self signed wildcard security certificate for apache or nginx), even though it says WILDCARD in the title, it clearly shows how to make one that is not a wildcard certificate

It may be true that tutorials teaching you how to create and install a self signed security certificvate are everywhere, this one here i put for my own purposes, because i use this one that i wrote as copy and paste instructions for speed. I added things to explain why we are doing everything

Here we will discuss installing a self signed certificate (Acting as your own certificate authorite) …

If you are not sure if what you need is a self signed certificate or a proper secure certificate signed by a certificate authority, you can see the difference here

In this tutorial, we will install a self signed certificate on a debian squeeze machine. Please keep in mind that 1 certificate can be installed per IP address. If you need to install more certificates for more hosts, please have a look here.

We are sssuming you have apache installed, other web servers have different installation instructions, but the generation procedure remains the same

The steps we will take are

A- get the system ready
B- Create a private key
C- Create a certificate signing request from the private key
D- Create a certificate from the certificate signinig request
E- Install the certificate and the private key
F- Decrypt the private key (Optional)

A- get the system ready

1- Install openssl

On my debian system, this is done with the following command, on other systems, the installer may be different

apt-get install openssl ssl-cert

2- Create a directory we can work in

mkdir /etc/apache2/ssl

B- Create a private key

Creating a private key is as simple as

1- Go to our working directory

cd /etc/apache2/ssl

2- To create a private key, Issue the command

openssl genrsa -des3 -out myprivate.key 4096

A password of your choice is requiered (You must enter it twice).

You will then have a private key in the file myprivate.key

This is the encryption key for your private key, and even though this is an encrypted private key file, this key should never be shared with anyone. Since if you do decrypt it in the optional step below, and if it is ever shared with anyone after it is decrypted, they can create a certificate just like the one we are creating and fool a visitor into thinking they are on the correct website (in case of a man in the middle attack for example).

C- Create a certificate signing request from the private key

The certificate signing request is the file we normally give to a certificate authority so that they can create a certificate for us, but in this case, we are the certificate authority (Self signed certificate), we will therefore create a Certificate Signing request and sign it ourselves

1- To create a file containing the Certificate signing request data, all we need to do is issue the following command

openssl req -new -key myprivate.key -out signingrequest.csr

You will now be asked for (Keep your eyes open for the common name since it is the most important)

* Your Pass Phrase, the one you chose for the private key (To create a request from a private key, we need the decrypt and read the private key)
* Country Code (US), State…, CITY, Organisation Name, Organisation Unit
* Common name, and this is the most important, this is either your domain or sub domain, if it is your domain, do not add WWW and enter example.com if it is a sub domain enter subd.example.com
* A Challenge password of your choice

* Enter anything into the optional company name.

We will now have 2 files in the directory, our private key (myprivate.key) and a certificate signing request (signingrequest.csr), we have 2 passwords, the private key’s encryption password and the signing request’s challenge password

D- Create a certificate from the certificate signinig request

openssl x509 -req -days 3650 -in signingrequest.csr -signkey myprivate.key -out mypublic.crt

Now, you will be asked the the Private Key’s pass phrase, the first password, Again to decrypt the private key so we can create a certificate.

We will now have 3 files in the directory, our private key (myprivate.key) and a certificate signing request (signingrequest.csr), and out certificate file (mypublic.crt)

E- Install the certificate and the private key

At this point, we have a public key (mypublic.crt), and a private key (myprivate.key), we can now install those on apache, and start using our certificate.

To install certificate on apache, we must

1- Enable mod ssl on apache, this is done with the command

a2enmod ssl

2- Make sure Apache is listening on the SSL port (443 by default)

On a debian system, you will need to verify the file ….

3- Fix the host’s config file to use the certificates

How this is done depends on how your system defines websites in apache, The easiest way to do this on a debian system is to copy the file mysite from the /etc/apache2/sites_available folder into a file called mysite_ssl (The mysite file could be called anything like polosite.com.cfg), then open the new file for editing, change the Virtual Host Line at the top to <VirtualHost *:443> (You may also change the 8 with your IP address) then scroll down to the end of the new file, and right before the end of the Virtual Host, add the following lines

<VirtualHost *:443>
...................
...................
SSLEngine on
SSLCertificateFile /etc/apache2/ssl/mypublic.crt
SSLCertificateKeyFile /etc/apache2/ssl/myprivate.key
</VirtualHost>

4- Restart apache server

On a debian system, the apache server is restarted with the command

/etc/init.d/apache2 restart

IMPORTANT: Now, when you restart the apache server, you will be asked for the password you chose when you created your private key, the next step below will make apache not ask for a password, but will also create a risk if your unencrypted key should fall into a malicious user’s hands. Worth mentioning that most websites do decrypt the private key, then make sure it is in a directory only root user can access, again, by default your private key is encrypted, decrypting it means you need to take very good care of it.

F- Decrypt the private key (Optional)

openssl rsa -in myprivate.key -out myprivate.key.insecure
mv myprivate.key myprivate.key.secure
mv myprivate.key.insecure myprivate.key
  

Now, restarting Apache should not ask you for a password

G- Working with browsers

Firefox will allow you to add this certificate to it’s memory, whenever this certificate is presented for this hostname, Your modified Firefox will consider it a valid certificate

Hard drive vibration and heat dissipation resolved

Update

In order to be able to stack disks outside computer cases, the solution i found was this
I got some of those grills used for paint, bent them so that they can hold the 3.5 inch hard drive through sc4rews, then i had a 12cm tube like aluminum tube into 1.5 inch pats, then glued the Sorbothane pads to the aluminum, then i duct taped the grills (that hold the disks) to the aluminum frame, and now i can stack them above each other.

Please be aware that vibration from disks affects other disks VERY considerably, i have had unbelievable failure rates with disks before this, the disk transfer rates also drops drastically when the vibration from other disks is high.

You might say that disk manufacturers claim that disks are relatively vibration resistant, if you dig deeper, you will find that vibration caused by a spinning disk is an exception to this, the type of vibration other disks will introduce.

Because i have many terabytes of files and backups, and because every PC i have has at least 4 hard drives attached to it, the vibration kills the performance of the hard drives, when you have 4 hard drives in a case, every hard drive is affected by the other three hard drives.

What i did was that i got a pack of vibration isolation pads, some metal grills that i use in suspending the disks (so that they can get better ventilation) (grills like the ones you could use in barbequing).

My hard drive failure rate went down considerably, actually, i have had no hard drives break down since i implemented this solution (I used to get plenty of failures).

So, the bottom line is, turns out the ONLY MATERIAL that isolates the vibration well enough (Actually so well) is called “Sorbothane Vibration Isolation”.

Anyway, here are SOME of the photos i took to demonstrate that.