command line fun – math

Posted on March 1, 2016 by DocDock

What is nice about shell scripting on Linux is that the language is just as complete as most of the compiled languages.  What elements makes up a pretty good computer language.

  • variables
  • assignments
  • conditionals
  • statements
  • flow control
  • expressions
  • functions

These elements are used in all computer languages from Lisp to C, and they all exist in bash script as well.  Some of these elements also exist in windows batch files but not all.  In Windows functions are a bit rudimentary and there is no math.

There are a few ways to implement math in bash scripts.  One of the methods that used to be used a long time ago is the external expr command.  This command will evaluate integer expressions.  The easiest way to see what this command can do is to try it on the command prompt.

> expr 3 \* 3
9

The reason for the backslash is because we need to escape the asterisk from the shell.  We need to do that for any character that could be interpreted by the shell.  If you are using parenthesis then this has a tendency to make equations look pretty horrible.

> expr \( 3 \* 4 \) / 2
6

Even if you can stand to look at your equations like this, you do have to be careful to include spaces between each element, failure to do so will yield the ever so helpful syntax error message.

There is really no reason to use this command any longer as there are easier ways to do it in shell itself.  The shell offers a easy syntax which allows you to enter the equation in a much more natural way.  Simply enclose your equation with inside of a pair of parenthesis.

$(( expression goes here ))

> echo $(( 5 * 17 / 7 ))
12

Not only is this easier to read, requires less care with respect to spaces but is also faster.  It is faster as the overhead of external command is no longer necessary.

Unfortunately there is a small drawback for both of these methods of evaluating expressions – they only support integer math.

In those cases where you really need floating point math you will have to use an external program.  The easiest choice is to use bc.  It is possible set the number of decimal places using the scale variable.

echo $(echo "scale=4; 1 / 9 " | bc )
.1111

This actually does the trick and if you are looking a specific number of decimal places this is perfect.  If you are not looking for a specific number of decimal places, it is easier to use the “-l” option in conjunction with bc.  This will load the mathlib but more important it will set the number of decimal places to 20.

echo 1 / 9 | bc -l
.11111111111111111111

This is actually really easy to do simple equations but unfortunately it is not possible to use any functions in your equation.  This makes sense as the functions are local to the shell while bc is an external program.

Posted in Command line, programming | Tagged , | Comments Off on command line fun – math

command line fun – windows flow control

Posted on February 21, 2016 by DocDock

Windows batch files are not nearly as rich as some languages (perl or python) and certainly not as rich as the shell scripting.

Not all of the familiar control structures exist but it is possible to create some as necessary and there are definitely enough to do write some batch files.

if / then / else

The most basic element that most developers use in their batch files is the if/then statement.  With this, you can compare strings or numbers and execute blocks of code or goto a different part of the script.

Just like C, the syntax of the if then actually drops the entire “then” from the statement.

@echo off
set a=4

if %a% equ 4 (
echo a=4
)

As the name implies, it is also possible to have an else statement added to our if clause.

@echo off
set /a a=4

if %a% equ 4 (
echo a=4
) else (
echo a is not equal 4
)

It should be obvious that left and right parenthesis are used to create the block of code to be executed.  The only thing not obvious about this statement is that the actual structure of the statement must be exactly as seen above.

“) else (“

The batch interpreter is very limited and does not allow you to add more white space, in the form of new lines, around the else statement.  This may be frustrating for someone who likes to format their code in a very specific manner but once you are aware of this short coming it is easy to work around.

There are six comparison verbs that can be used in the if/then/else tests.

EQU equal
NEQ not equal
LSS less than
LEQ less than or equal
GTR greater than
GEQ greater than or equal

small gotchas

Not every script necessarily does math, the numbers may not be important at all for what you do.  If you do not indicate that you are doing numeric calculations, then the batch interpreter thinks that you are doing string addition not numeric calculations.

set a=4
set a=%a%+1
echo a=%a%
> a=4+1

However, if you plan on doing math, you have to make sure that you add the “/a” to the set command.

set a=4
set /a a=%a%+1
echo a=%a%
> a=5

 

other verbs

The batch interpreter, just like the bash shell, has some additional support for checking that files do or do not exist.

@echo off
IF EXIST "temp.txt" (
    ECHO found
) ELSE (
    ECHO not found
)

To negate the test, simply add the key word NOT in front of exist.

@echo off
cd /d c:\temp
if NOT exist "%1" echo file %1 exists

for loop – processing files

There are a number of different types of actions that are supported with the for loop.  The for loop doesn’t seem quite as flexible as the for loop in Linux but powerful enough to gather up a list of all files in a directory to be processed.

@echo off
SET count=1
FOR /f %%G IN ('dir /b *.png') DO (call :forloopfunct"%%G")
echo here
GOTO :eof

:forloopfunct
echo %count%:%1
set /a count+=1
GOTO :eof

echo i am now here

This example will loop through all files in the current directory that have the extension of png.  The syntax for windows batch files is somewhat different than in Linux.  For loops in windows batch files do not have a block of commands associated with the for loop itself.  If you want to execute a number of commands as part of the for loop, you need to have a similar construct, where the block of commands are put into a function that is called.

This script is a complete script despite what looks to be missing labels. It appears that the eof label in the subroutine is essentially a return statement.  The same is true earlier in the script, the first “goto :eof” appears to also be a return statement.  The final echo statement in the script is never executed.

There are a number of qualifiers that you can use in a for loop.

qualifier Description
/F List of all files in current directory.
/R Recursive search through all sub-directories
/D List of directories
/L For loop over a range of numbers

for loop – over a range of numbers

It is possible to do a more standard for loop, from a starting number stepping up to a final value.

@echo off
 FOR /L %%i IN (0 2 100) DO (
 echo %%i
 echo hiya
 )

This small example starts at zero and counts up by two until equal one hundred.

The syntax is slightly different than the other for loops described.  It is also possible to call a subroutine for each element but in this type of a loop it is possible to have a block of commands to be executed with iteration.

 

pseudo case statement

There is no switch statement in windows, but it is possible to create one using a goto statement and clever use of labels to create a switch statement.

@echo off
SET I=1

GOTO LABEL_%I%
:LABEL_0
ECHO i equals 0
GOTO END_SWITCH

:LABEL_1
ECHO i equals 1
GOTO END_CASE
:END_CASE
echo program finished

 

functions

The best way to keep your code clean and easy to understand is to modularize it into functions.  Windows also supports this by allowing you to create subroutines.

It seems to be an odd to me, but Microsoft implemented this much like basic from years ago.  To call a subroutine you have to actually say “call” and the name of the subroutine.

@ECHO OFF

:: script global variables
SET day=23 
SET month=11
SET year=2015

::main
call displaydate %day% %month% %year%
goto :eoj

:displaydate
echo %year%%month%%day%
goto :eof

 

Other tricks

It is possible to run the normal dos commands and have the output assigned to a variable. Simply surround the command with percentage symbols and treat it like a variable and the output from the command will be expanded and assigned like a variable.

@echo off
set currentdirectory=%cd%
echo the current directory is %currentdirectory%

It is possible to save the current working directory in the manner described above.  This allows the script to change directories and easily find its way back to where it started.  However, there is an even nicer way to do that particular trick.

Microsoft has two commands that will save the current working directory while changing to a new directory.  This is implemented as a LIFO stack, so it is possible to always return to the last directory that you were in.

@echo off
echo starting script

echo amount of space used in temp directory
pushd c:\temp
du -h .

echo " "
echo list of users who had logged into this machine
pushd c:\users
echo now i am in %cd%
dir /b /q *.

echo " "
echo information about disk in general
if exist "c:\program files\windows media player" (
echo found it
dir "c:\program files\windows media player\wmplayer.exe"
) else (
echo cannot find windows media player
)

:: pull directories off stack
popd
echo now i am in %cd%
popd
echo now i am in %cd%

It is possible enhance your windows machine by downloading or writing comand line utilities.  One that might be interesting is the windows du, similar to the unix du but runs on windows.

windows utility du
https://technet.microsoft.com/en-us/sysinternals/du.aspx

Posted in Command line, programming | Tagged , | Comments Off on command line fun – windows flow control

Why do they hate us … by the IRS

Posted on February 17, 2016 by DocDock

I have a tendency to make the same mistake every year. I take in a pile of disorganized receipts and numbers to someone who will be helping me do my US taxes. It is usually about this time that I get to hear about yet another new requirement or a changing of the rules and then I think “IRS you rotten bloobity blabity group”.

I should really be thinking you §$%&/%$ elected politicians who have been lobbied either externally or internally about changing the rules to make sure that some innocent freebie does not go untracked.

This year while providing a stack of information I received the question about the “Form 8938 (Statement of Specified Foreign Financial Assets).” Huh?

This filing requirement would be triggered if you had “specified foreign financial assets” in an amount greater than 200,000 on the last day of the year (12/31/14) or more than 300,000 at any time during the year.

This was probably brought up by those kind people who decided we should be filling out the Foreign Bank and Financial Accounts (FBAR) report.  It is a wonderful little form that lists the name, address, bank account details, and your personal details – for every single account you have access to.  Oh, just to make it really series, did we mention the penalties.

For willful violations, the penalty may be the greater of $100,000 or 50 percent of the balance in the account at the time of the violation…

Perhaps the blame shouldn’t be put at the feet of the Internal Revenue Service but rather the politicians.  Yet at tax time, it is hard to make the distinction.

Posted in Soapbox | Comments Off on Why do they hate us … by the IRS

finding more IT staff

Posted on February 14, 2016 by DocDock

President Obama is apparently planning on asking the Congress to allocate four billion dollars to increase the number of people who have access to computer science.  This could be indeed a well intentioned plan to ensure that the most people possible have access to computer technology and are ready for new challenges for the new millennium.

If you are a cynical type, it could be a plan designed to pander to the large corporations who can never seem to get enough IT talent.

I am going assume a little of column A and little of column B.  A normal supply and demand curve shows the number of IT people available over a range of compensation against the demand of companies of IT people at various levels of compensation.

labor-supply

The problem is not necessarily the supply of technical staff as much as the supply at the level that the corporations wish to pay.

company-demand

Levels of compensation

The market will supply more people available by increasing the levels of compensation for the technical people.

fixed-labor-supply-1fixed-labor-supply-2

Like magic, if technical degrees seem to be the ticket to well paying jobs, men and women will flock to the universities to get their degree.  When the levels of compensation are higher, then the supply of available applicants goes up.

There are of course other reasons why the ranks of IT don’t always swell with people.

  • expected to work longer/harder/evenings and don’t complain
  • can be a solitary profession
  • need to be available 24/7
  • weekend work

These job attributes are not necessarily associated with every technology job but stereotypes exist because of the kernel of truth.  These do accurately describe some tech positions depending on the type of job and the company who employs you.

These reasons will help to eliminate people or groups of people depending on their situation.  These might not be much of a factor to recent graduates but it might to people slightly older or someone who has a family.

These factors actually may reduce the level of perceived compensation in the eyes of some applicants considering IT careers.

These attributes would be a form of negative compensation just on their own, but it doesn’t help when other factors or companies conspire to lower the compensation or keep it down.  Yet, this type of direct influence isn’t the only way to keep wages down.

Legal exemptions

The cynic in me believes that perhaps a bit of lobbying was going on to exempt some of those tech workers fro the Fair Labor Standards Act.

The FLSA establishes minimum wage, overtime pay, recordkeeping, and youth employment standards affecting employees in the private sector and in Federal, State, and local governments.

http://www.dol.gov/whd/flsa/

It is obvious why companies would prefer to have a group of workers who they don’t have to pay overtime to.  That isn’t to say that some companies don’t pay overtime, but if this little goodie is tucked into the law, a company would literally be a bit crazy to not take advantage of a legal method of keeping wage costs down.

H-1B Visa’s

The most market driven way to decrease the wages in general would be to somehow increase the number of IT specialists.  The local developers and tech workers would never increase their numbers with the promise of lower compensation.  There is actually a more direct way, simply import workers who can perform the jobs but do it in such a manner that they have less rights than a local worker.  The H-1B visa is the answer.

  • No automatic conversion to permanent residence status
  • H-1B visa holders can be laid off at will
  • company is able to discharge of an H-1B worker if the employer wishes to hire a U.S. worker

The extra bonus is that when the visa holder is discharged he is given 10 days to get a new valid visa or go home.  These three points would form a pretty good incentive for the H-1B visa holder who wishes to remain the USA to keep in line.

These visas must be very popular as Microsoft [1] [2] [3], Intel [4] [5], Facebook [6], just to name a few companies are allegedly trying to get larger numbers of visas and to “reform” the visa system.

Is the H-1B visa program being used to get experts that are not available any other way, or this is just another way to get cheaper labor than is available locally.

Is there really a shortage

If you cannot find some IT talent for love nor money then there may be a shortage of available talent.  The question of how do you decide what skills are needed for your positions.  I just read an interesting piece from the IEEE which actually has an interesting view on the lack of tech workers.

To be honest, indeed when a language or technology becomes the hot commodity it can be difficult to find people with a lot of experience in that technology.  However, the job description cannot be as follows.

Wanted senior software engineer, intimately familiar with Web Services, Fortran, Lisp, Eiffel, 6502 assembler and COBOL who is comfortable writing their own proprietary operating system that is scalable from the Atmel ATtiny13 up through a mainframe.  Will require no assistance in our development environment and has 15 years experience as both a investment banker and gourmet chef.

Those people already work for you, perhaps your competition or don’t exist at all.

Perhaps President Obama is not necessarily focusing on the right goals for the future of IT.  Perhaps the goals should be to remove the forces that distort the invisible hand of the market, rather than a half hearted attempt to encourage more people into the profession.

 

Wage fixing cartels
https://pando.com/2014/03/22/revealed-apple-and-googles-wage-fixing-cartel-involved-dozens-more-companies-over-one-million-employees/

Text of the Computer Professionals Update Act
https://www.govtrack.us/congress/bills/112/s1747/text#

Overly high expectations when looking for candidates
http://spectrum.ieee.org/podcast/at-work/tech-careers/why-bad-jobsor-no-jobshappen-to-good-workers

The uses and abuses of H-1B visa’s
http://www.cringely.com/2012/10/23/what-americans-dont-know-about-h-1b-visas-could-hurt-us-all/

Posted in Soapbox | Comments Off on finding more IT staff

command line fun – flow control

Posted on February 11, 2016 by DocDock

Shell scripting is like any other programming language as far as control statements go.  To be honest, all your favorite control statements can be whipped up using either if/then statement or a loop.

The good news is that we don’t need to create this from scratch but can instead use all the same control statements as most modern languages.

IF / THEN / ELSE

The general statement is the same as other languages but the actual comparison verbs are slightly different.

Operator Description
-z string is null
! -z string is not null
== test for strings being equal
!= test for strings not equal
-eq numeric equals
-ne numeric not equals
-gt numeric greater than
-ge numeric greater than equals
-lt numeric less than
-le numeric less than equals

The comparison verbs are different depending on whether comparing numbers or strings.

#!/bin/bash
A=1
B=4

if [ $A -eq $B ]
then
  echo $A
  echo A equals B
else
  echo $A $B
  echo A not equal B
fi

WHILE

The while construct simply does a comparison at the beginning and executes the body if the statement is true.  The comparison verbs are the same list of verbs as seen above for the if/then/else statement.

Just like in C, you can use either break to exit the loop completely, the continue statement causes the program control to pass to the conditional tests.

#!/bin/bash
A=1
B=4

while [ $A -le $B ]
do
  echo hi $A
  A=$(($A + 1))
done

FOR LOOP

The for loop, when it can be used, is perhaps one of the best control structures to use.  It could be a personal hangup of mine, but the advantage is that the initialization, test and increment is all conveniently located in one spot.  This makes it hard to forget to setup one of these three crucial components.

#!/bin/bash
for (( idx=1 ; $idx < 9 ; idx=`expr $idx + 1` )) ; 
do 
   echo "some neat message here $idx" 
done

Oddly enough, I almost never use the for loop in this manner.  The much more convenient way of using it is almost like some sort of set operator or iterator.  Rather than counting something, the for loop is used to iterate through a small collection of items.

#!/bin/bash
VALS="1 2 3 5 8 13 21"
for i in $VALS
do
   echo $i
done

This particular example is pretty unimaginative and unrealistic.  The better case is to collect some data from a file or file-system and then to process each one.  This could be to run a special program against each file individually, or to perform some special steps one at a time.

#!/bin/bash
CSVFILE=output.csv
FILES=`ls -1 control-data-2016*`
for i in $FILES
do    
   CNT=`wc -l it.sh | sed -e 's/ .*//'`
   echo $i,$CNT >> $CSVFILE
   gzip -9 $i
done

This small script is probably not the simplest way to count the number of lines in a file and prepare the output for a comma delimited file but it shows one iterative method of processing files.

SWITCH STATEMENT

The switch statement is much nicer than a really big if/then/else block.  Simply separate each case block with two semicolons.

#!/bin/bash
argument=$1

case "$argument" in

  start) 
  echo starting service 
  ;;

  stop) 
  echo stopping service 
  ;;

  *) 
  echo unknown command $argument encountered 
  ;;

esac

Depending on the problem, I am sure there are a number of uses for this structure but I don’t normally have much use for it.

The main reason is due to quite a few other operators that can be used with the if/then/else or while structures.  It makes it easy to verify that files or directories exist and are readable or writable.

Operator Description
-f file exists
-d directory exists
-w if file is writable
-r if file is readable
-x if file is executable

 

Posted in Command line, programming | Tagged , | Comments Off on command line fun – flow control

the joy of non-standard graphic formats

Posted on January 31, 2016 by DocDock

It was a wild and carefree time in the nineties.  Partying with friends, going on vacations, as well as traveling to exotic locations.  Some of my friends had not only gone on vacations but they had Kodak create some CD’s with their holiday photos. It was pretty cool at that time so I went out and did that as well for some of my photos.

Fast forward to today.  For the most part nobody gets their film developed nor converted to photo CD’s, we simply copy them from the camera to our computer and cut out some of those obsolete steps.

While going through my desk drawers I found some of these old photo CD’s. It was only at this point that I realized that the solution at that time was fine but none of my current programs can deal with this proprietary PCD as a photo format.

This format, although proprietary, can be manipulated by a number of different programs.  Some of the choices available such as Irfanview for Windows or iPhoto for the Macintosh.  This is not the limit of available choices just a few of those that can be purchased.  I didn’t want to purchase a piece of software to convert seven CD’s never to be used again.

Yet, I was lucky enough to find some source code which can only convert from PCD to JPEG but that was exactly the conversion that I was looking for.

PCDTOJPEG

The utility, pcdtojpeg, is a command line program which is distributed as three source files.

Simply get the latest source at http://sourceforge.net/projects/pcdtojpeg/files/latest/download.

I didn’t compile this on Windows or for the Macintosh, just on Linux.  Unlike a lot of Linux projects there was no makefile nor complicated shell scripts.  Just a single command will compile both source files into the binary.

> g++ main.cpp pcdDecode.cpp -ljpeg -lpthread -o pcdtojpeg
> chmod 777 pcdtojpeg
> ./pcdtojpeg

pcdtojpeg, version 1.0.12 
Copyright (C) 2009-2015 Sandy McGuffog
JPEG compression code Copyright (C) 1991-1998, Thomas G. Lane.

Usage:  ./pcdtojpeg [options] file1 [file2]

Valid options:
-h            Print this message
-v            Verbose file information
-m            Process the file as monochrome
-D50          Process for a white balance of D50
-D65          Process for a white balance of D65 <default>
-q nnn        JPEG file quality (nnn range 1 to 100 <100>)
-b n.n        Brightness adjustment (n.n range -2.0 to 2.0 <0.0>)
-r n          Highest resolution to extract (n range 0 to 5):
                 0 - Base/16 (128 x 192)
                 1 - Base/4 (256 x 384)
                 2 - Base (512 x 768)
                 3 - 4Base (1024 x 1536)
                <4 - 16Base (2048 x 3072)>
                 5 - 64Base (4096 x 6144)

I actually relish the command line, so using this utility was perfect.  To use this program, I created a small shell script which takes all PCD files from the current directory and converts them to JPEG.

#!/bin/bash
convert()
{
   ./pcdtojpeg -q 100 -r 4 $1 ${1}-4.jpeg  &
}

LIST=`ls -1 *.pcd` 

for i in $LIST
do
   convert $i
done
Posted in Command line, Setup From Scratch | Comments Off on the joy of non-standard graphic formats

command line fun – logs and logging

Posted on January 28, 2016 by DocDock

Before writing this blog entry, I never considered the actual similarities between the Linux shell and the dos shell. I shouldn’t have been too surprised as the similarities have less to do with the operating systems than with the general theory.

The shell is all about text. Command line programs accept text as input or generate text as output.  When things go well the text goes to standard output, but when things don’t go well then the messages go to standard error.

Output on the command prompt isn’t entirely obvious when it is printed.  The content may give an indication if it says something like “syntax error” but there is a better way to determine the real problems.  The output streams are separate and can be redirected either separately or together to a log file.

All the output is displayed together in the terminal window but can be split into its composite parts.  In Unix each program inherits three standard streams, which is how the program can then split the output into either standard output or standard error.

Standard IO Streams

Fileno Name Description
0 STDIN Standard input
1 STDOUT Standard output
2 STDERR Standard error

For example, the directory listing output of the ls program is sent to standard output but the error is sent to standard error.

sam@asus:/tmp/testarea$ ls -ltrR
.: total 388
-rw-r--r-- 1 sam sam 146962 Nov 7 23:54 Steigenberger Airport_Kinder_009.jpg
-rw-r--r-- 1 sam sam 92444 Nov 7 23:54 Steigenberger Airport_Kinder_010.jpg
-rw-r--r-- 1 sam sam 148526 Nov 7 23:54 Steigenberger Airport_Kinder_031.jpg
d--------- 2 sam sam 4096 Nov 7 23:55 private
ls: cannot open directory ./private: Permission denied 
sam@asus:/tmp/testarea$

The output can be redirected to a file by using the greater than “>” symbol.

sam@asus:/tmp/testarea$ ls -ltrR > listing.txt
ls: cannot open directory ./private: Permission denied 
sam@asus:/tmp/testarea$

Yet, because the error output is not redirected, it is still displayed in the terminal window.

It is possible to only redirect the error messages to the error file.

sam@asus:/tmp/testarea$ ls -ltrR 2> error.txt
.: total 388
-rw-r--r-- 1 sam sam 146962 Nov 7 23:54 Steigenberger Airport_Kinder_009.jpg
-rw-r--r-- 1 sam sam 92444 Nov 7 23:54 Steigenberger Airport_Kinder_010.jpg
-rw-r--r-- 1 sam sam 148526 Nov 7 23:54 Steigenberger Airport_Kinder_031.jpg
d--------- 2 sam sam 4096 Nov 7 23:55 private

Yet, it is easy to combine the output of both streams into a single output file by adding “2>&1” to the end of the command.  This will then redirect all standard error output into the standard output.

sam@asus:/tmp/testarea$ ls -ltrR 2>&1 output.log
.: total 388
-rw-r--r-- 1 sam sam 146962 Nov 7 23:54 Steigenberger Airport_Kinder_009.jpg
-rw-r--r-- 1 sam sam 92444 Nov 7 23:54 Steigenberger Airport_Kinder_010.jpg
-rw-r--r-- 1 sam sam 148526 Nov 7 23:54 Steigenberger Airport_Kinder_031.jpg
d--------- 2 sam sam 4096 Nov 7 23:55 private

Special output locations

It is possible to decide which output is important.  In some situations it may be the normal (standard) output while in perhaps for production situations it is only any errors that occur that are interesting.

On both Linux and dos it is possible to easily throw away the unnecessary output.  The virtual garbage can is called the null device.  So, to eliminate unneeded output simply redirect the output to the null file.

The null file is called /dev/nul on linux and NUL on windows.

Ignore standard output example

Linux
   ls -l file.xxx > /dev/null

Windows
   dir file.xxx > nul

Ignore error example

Linux
   ls -l file.xxx 2> /dev/null

Windows
   dir file.xxx 2> nul

Other output methods – tee

One of the really neat things about the Unix world is the wealth of command line programs which simplify daily operations.  One of my favorite command line programs is tee.  Simply run some program and pipe the output through the tee command.  The output will be both displayed both on the screen but also logged to the file given.

ie.ls -ltr | tee mylogfile.log

This is ideal when you wish to run some program or script on the command line and see what is being printed but still wish to save the standard output to a log file.  But if the error output should also be saved, then simply redirect the standard error output to the standard output stream.

There is no standard program or utility that is similar to the Linux tee command in the windows world but there is a similar utility that can be downloaded for windows to supplement windows scripts.

Other output methods – script

It is all well and good to redirect output.  It is possible to see what the output was or to get a small extract of the errors that occurred, but there is a better way to see not only the run of a single command but everything that occurred at the command line for a certain period of time.

The Linux command “script” will save every key pressed and every character or message that was output to a output file.

If the command is run from the command prompt without any parameters then all output will be saved into a file called “typescript”.  If a filename is given as a parameter then all output will be saved to that file.  All key presses or output will be stored into the output file until you terminate the script command.  This is done by either typing “exit” or with a control-d.

Posted in Command line | Tagged , , | Comments Off on command line fun – logs and logging

can’t shutdown my §$%&/#+ Windows PC

Posted on January 27, 2016 by DocDock

Most people would likely say that their work computer isn’t fast enough, but then which computer is.  I am not actually unsatisfied, my laptop is good enough for compiling, using secure shell, mail and word processing. The company actually does have a pretty good VPN setup to allow me to work remotely.

Despite the fact that everything works well, I do have one very tiny little complaint.

The only problem is that I cannot properly shutdown when I am working remotely, it just hangs.  I have tried to shut it down by briefly pressing the power button sometimes that helps, sometimes it causes the computer to hibernate.

It is a bit difficult to get this corrected by IT as it only happens when I am connecting remotely.  When I am at the office and IT looks at it, it never misbehaves.  I had someone from IT look into this and they cannot find any reason why my laptop cannot shutdown.

I used to simply power the computer off without shutting down at all.

Command line to the rescue

Windows actually provides the shutdown.exe command which actually works much better than selecting shutdown from the start button.

option description
/s Schedules a shutdown of the computer after a few minutes.
/r Shuts down the computer and then does a restart.
/a Aborts the current scheduled shutdown
/f Force applications to stop without informing the user.

It isn’t the shutdown command itself that is so powerful but the force option.  I suspect that the software that was used to build the vpn is the problem despite having terminated the connection.

I don’t want to let windows make all of the decisions, I close all of the applications that I am using but now I can call shutdown to gracefully power down.

shutdown.exe /s /f

Posted in Command line | Tagged , | Comments Off on can’t shutdown my §$%&/#+ Windows PC

securing your computer – veraCrypt

Posted on January 20, 2016 by DocDock

In a previous article I described how you can use ecryptfs to secure your partition on your computer.  It was a cool solution that but it was only available on Linux and all of the setup was done from the command line.

VeraCrypt is a encryption program which can be used to encrypt a partition or to create a file container which becomes a encrypted file system.  The file container unlike a partition can then be copied like any other file to other disks, to another USB stick or even to another computer.  In addition to supporting both partitions and file containers it also provides a small GUI to assist in setting up partitions and for mounting them.

You can probably download a copy of VeraCrypt from a number of sources but the best would be from their webpage codeplex.com.

Verify downloaded file

Like any other open source software it is a good idea to verify no problems during the download and that the file has not been tampered with.  Linux provides the sha256sum program to generate the hash that can be matched against provides list of hashes provided by VeraCrypt.

In addition to the file, you will need to either download the checksums or look them up from the site.

veracrypt-1.16-sha256sum.txt
aafacca9a600af5b8d66387718c984b8655905f72370bbd772baf90e57e85b7e VeraCrypt Setup 1.16.exe
f5c70ad7ea8dd660f62b9162f745728ccfad1d00e74b3a4eedccf6c3d92eb43f VeraCrypt User Guide.pdf
bfe147cb4c0a0e8ab47fa71ae0d3eec825f49548246da6e4a75a7b9b6250d78c VeraCrypt_1.16.dmg
6861e79eb7e662330fa2a304061ebfb6a56929a78d8f4841ed0449a553257e7a veracrypt_1.16_Source.tar.bz2
0a1c6b8165d78be62623194178a109bdd8f8b4dbcb6c24d8b15eba629f99ddaf veracrypt_1.16_Source.zip
81afbde794ea8ff426f4b5ecfe72269fbdc9b99bb759f42eaf54936d1a7dd1ba veracrypt-1.16-setup.tar.bz2

Simply run the sha256sum with the downloaded file as the first parameter.

> sha256sum veracrypt-1.16-setup.tar.bz2
81afbde794ea8ff426f4b5ecfe72269fbdc9b99bb759f42eaf54936d1a7dd1ba veracrypt-1.16-setup.tar.bz2

It just isn’t quite that simple in the windows world as no program for calculating these hashes are shipped with the operating system.  Yet, it is easy enough to find one, install it and then use that.

For windows I downloaded the project quickhash.  This is a really easy program that can calculate the hash for text, files, disks and a whole lot more.  They even ship documentation.  If you are a windows user, this just might be the package for you.

However, from the user manual provided there may be some confusion over the name.  It is possible that other developers have also developed their own program with the same name.

Install Windows

VeraCrypt has a standard windows installer, which walks you through the entire process.

winVeraCrypt1

The VeraCrypt license that needs to be accepted

 

winVeraCrypt2

Install dialog

 

winVeraCrypt3

Another Install dialog

 

winVeraCrypt4

Dialog showing the current status of the install

winVeraCrypt5

Install Linux

The tar file containing VeraCrypt has been bzipped, and so you need to unzip it first.  The bzip2 program is not as common of a choice for compressing data.

There have been a number of different analysis’s between gzip and bzip.  The analysis can be summarized as gzip compresses faster but bzip has superior compression.  The bzip speed/compression issue may be less and less important as the Linux kernel discontinued using it march 2013, it is possible that this will be the beginning of a trend.

Command Description
bzip2 -d file.bz2 unpack the contents of the archive and delete the archive once that has been done.
bzip2 -dk file.bz2 unpack the contents of the archive and keep archive.

The installation is just running a shell script.  Simply run the script that is appropriate for the version of your operating system.  They also provide two command line choices as well.

> bzip2 -d veracrypt-1.16-setup.tar.bz2 
> ls -ltr
-rw-r--r-- 1 dock   dock    16865280 Jan 13 23:28 veracrypt-1.16-setup.tar
> 
> tar xvf veracrypt-1.16-setup.tar
> veracrypt-1.16-setup-console-x64
> veracrypt-1.16-setup-console-x86
> veracrypt-1.16-setup-gui-x64
> veracrypt-1.16-setup-gui-x86
>
> sh veracrypt-1.16-setup-gui-x64
>

Once you run this script, a few dialog’s will appear, simply click through the dialogs and enter the root password when prompted.

verycrypt-setup

Install dialog

 

verycrypt-eula

The VeraCrypt license that needs to be accepted

 

verycrypt-uninstall

A reminder on how to uninstall VeraCrypt

 

verycrypt-install

Installation log output

It takes only a few seconds to install the software, and then you are ready to run the software.

Setup

The documentation for veraCrypt is quite complete.  It would be difficult to explain in few pages what they cover in quite a lot of detail in over 150 pages.

It is probably enough, to briefly describe the two different types of encrypted disks that would be created.  The first is actually not really a disk but really a very small file system saved as a single file on your normal file system.

When this file is mounted with veraCrypt it becomes just like any other disk drive that can be written to.  Because it is a single file, it could easily be copied to a USB stick, to another computer or to some sort of Internet storage device (cloud).

This is actually the best choice if you are not very comfortable with the lower levels of the disk drive such as partitions.

The second type of disk would be an actual disk partition.  Instead of creating a file on an existing hard disk, a partition is selected and encrypted.  This disk, when mounted, also behaves like any “normal” linux or windows file system.

When setting up your encrypted disk, you will be prompted for which encryption protocol amongst others things, but there is a rather harmless looking question that shouldn’t be overlooked.

verycrypt-setup7

Do you want to store files larger than 4gb.  It really depends on the type of data that you will be storing in this encrypted drive how you answer.  Most data does not get anywhere near this size.  A couple of exceptions might be some sort of raw video footage, or if you have ISO images from double layer DVD’s.

Another harmless looking question is if you plan on only accessing this encrypted disk from more than one operating system.

verycrypt-setup9

I suspect that quite a few Linux users still have a partition with a copy of windows that is used for some minor tasks.  Setting this option will let you use veraCrypt on other platforms. This makes your encrypted partition accessible from both operating system.  It can even be used as a method for transferring data between the two systems.

There is one small difference between veraCrypt and its predecessor truecrypt.  When formatting the encrypted disk you simply select the file system, however, it is now possible to select NTFS even from Linux.  Previously, the file systems available were only those supported by the running operating system. verycrypt-setup8

Limitations

The good news is that veraCrypt seems to be picking up where truecrypt left off.  The solution provides a nice GUI which makes it really easy to mount encrypted files or partitions for even the most casual user.

The only real limitation that I encountered was that veraCrypt said it was compatible with truecrypt volumes.  This might be the case, I did not have this experience.

Posted in security, Setup From Scratch | Tagged | Comments Off on securing your computer – veraCrypt

booting Linux from a USB stick

Posted on January 20, 2016 by DocDock

Most of the Linux distributions are downloadable as a ISO image which can be burned to a CD or DVD.  This is pretty convenient, download the distribution and create your own disks.

In the beginning floppy disks were used by everyone to transport data or install software.  Once the software was too big to fit, the floppy disks started to disappear and everyone moved to CD’s and DVD’s.  The same thing is now happening to the once so familiar CD and DVD drives.

They haven’t really been replaced by the much larger blue ray discs but have instead been replaced by fast network connections and USB sticks as USB sticks are small, fast, have no moving parts, and fit into your pocket.

In order to retain all of the same functionality, it became important to be able to use the USB sticks in the same way as the DVD.  USB sticks exceed the size of the disc’s without breaking a sweat.  The only missing piece was bootable USB sticks.

This was solved with the introduction of the isohybrid feature.  This is a reasonably small modification to the boot record, that allows USB sticks to be booted from BIOS just like a CD or DVD.   A few years ago if you wanted to get a Linux distribution to boot from the USB stick you had to run programs to make the conversion to your distribution and then copy it to the USB stick.

Now it is pretty common that a lot of Linux distributions when they create their ISO images already have this isohybrid format, especially the live discs.

There is virtually no effort involved to take one of these images and get it to boot off of a USB stick.  A single command will transfer the image.

dd if=myimage.iso of=/dev/sdX bs=1mb

This command will copy the image byte for byte over to the device /dev/sdX.

It is really important to make sure you give the proper device otherwise, you may be overwriting your main hard disk with the ISO image.  There are probably a lot of methods to determine what the device is, I either look at the logs with the dmesg command or the devices connected with the lsblk.

Log messages

> dmesg
 [ 5624.106378] usb 3-2: new high-speed USB device number 3 using xhci_hcd
 [ 5624.234666] usb 3-2: New USB device found, idVendor=05dc, idProduct=0300
 [ 5624.234672] usb 3-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
 [ 5624.234675] usb 3-2: Product: JUMPDRIVE GEYSR
 [ 5624.234678] usb 3-2: Manufacturer: LEXAR MEDIA
 [ 5624.234680] usb 3-2: SerialNumber: 0A4EEC090022451
 [ 5625.037188] usb-storage 3-2:1.0: USB Mass Storage device detected
 [ 5625.037382] scsi6 : usb-storage 3-2:1.0
 [ 5625.037549] usbcore: registered new interface driver usb-storage
 [ 5626.033751] scsi 6:0:0:0: Direct-Access     LEXAR    GEYSER JUMPDRIVE 1.00 PQ: 0 ANSI: 1 CCS
 [ 5626.034308] sd 6:0:0:0: Attached scsi generic sg3 type 0
 [ 5626.034387] sd 6:0:0:0: [sdc] 2014992 512-byte logical blocks: (1.03 GB/983 MiB)
 [ 5626.034935] sd 6:0:0:0: [sdc] Write Protect is off
 [ 5626.034952] sd 6:0:0:0: [sdc] Mode Sense: 23 00 00 00
 [ 5626.035486] sd 6:0:0:0: [sdc] No Caching mode page found
 [ 5626.035491] sd 6:0:0:0: [sdc] Assuming drive cache: write through
 [ 5626.038184]  sdc: sdc1
 [ 5626.039933] sd 6:0:0:0: [sdc] Attached SCSI removable disk
 [ 5629.732887] FAT-fs (sdc1): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!
 [ 5629.743731] FAT-fs (sdc1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.

Simply run the dmesg command and then do it again after plugging in the USB stick.

List of block devices

> lsblk
 NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
 sda      8:0    0 698.7G  0 disk
 ├─sda1   8:1    0   200M  0 part /boot/efi
 ├─sda2   8:2    0   128M  0 part
 ├─sda3   8:3    0 279.5G  0 part
 ├─sda4   8:4    0 393.9G  0 part /
 └─sda5   8:5    0    25G  0 part
 sdb      8:16   0 698.7G  0 disk
 ├─sdb1   8:17   0   9.5G  0 part [SWAP]
 ├─sdb2   8:18   0  46.6G  0 part
 ├─sdb3   8:19   0  50.2G  0 part
 ├─sdb4   8:20   0 451.8G  0 part
 ├─sdb5   8:21   0  47.5G  0 part
 └─sdb6   8:22   0  93.1G  0 part /home
 sdc      8:32   1 983.9M  0 disk
 └─sdc1   8:33   1 983.4M  0 part /media/dock/LEXAR MEDIA
 sr0     11:0    1  1024M  0 rom

Both of these methods let us know that there is a device /dev/sdc and that device has a single partition.

In my latest case, I used the following command.

> dd if=debian-8.2.0-i386-netinst.iso of=/dev/sdc bs=1m
> sync

The command sync has nothing to do with the copy but simply ensures that all I/O buffers are flushed.  This guarantees that the data is written out to the device.  This shouldn’t be necessary but it is a good practice to get into.

Don’t make the mistake of writing the ISO image to the device’s partition.  The boot record is at the beginning of the ISO image and must in the same location of the USB device otherwise the computer will not recognize the operating system and will hang.

Posted in Command line, Setup From Scratch | Tagged , | 1 Comment