Maven simple example

Maven was briefly described in my previous post.  In this post I will do a few small examples for actually using Maven.

One of the main things about Maven is the idea of the artifact.  The artifact is a file, which is usually a jar file.  This might be a jar that is a dependency or a jar, war or ear file that is generated by Maven.  All artifacts are described by three different key values.

  • artifactId
  • groupId
  • version

The artifactId is just a name.  The group id is also just a name but it is usually it is the reverse domain of the company.  The version number is just the version of that particular object.

Hello world – manual

The directory structure is fairly simple.  The program code will be under the src/main directory structure while the unit tests will be under the main/test directory.

mkdir helloworld
cd helloworld

mkdir src
mkdir src/main
mkdir src/main/java
mkdir src/test
mkdir src/test/java

The structure is simple but must be strictly adhered to.  The structure is well defined so the project object model (pom) file doesn’t need to contain much at all.

<project>
  <modelVersion> 4.0.0</modelVersion>
  <groupId> de.acmesoft </groupId>
  <artifactId> helloworld </artifactId>
  <version> 1.0.0 </version>
</project>

It is important to have the artifact keys in the pom file and they should be different than any other artifact keys.  This is important in case this artifact should end up in the repository.

The modelVersion field is refers to the internal structures for Maven.  In future releases of Maven this version may change.

The executable from Maven is mvn.  Simply pass in the phase that you are interested in and Maven will execute the lifecycle up to that point.

For this sample I could simply compile the class files or go one step further and generate a jar file.

#: /tmp/mvn mvn compile 
[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Building helloworld 1.0.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-resources-plugin:2.3:resources (default-resources) @ helloworld ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /home/dock/working/content/43-blog/maven/helloworld3/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ helloworld ---
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 1 source file to /home/dock/working/content/43-blog/maven/helloworld3/target/classes
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.974s
[INFO] Finished at: Sun Dec 11 23:27:27 CET 2016
[INFO] Final Memory: 10M/119M
[INFO] ------------------------------------------------------------------------

Running hello world

Maven creates all of it’s output files into the target directory.  This is true for both the normal class files as well as the jar files (for simple java programs) that get created.

If you want a quick test simply set the class path to point to the new jar file.

java -cp target/helloworld-1.0.0.jar com.acmesoft.helloworld

If you look closely at the pom file above you will see that it is inconsistent from the pom file. The package in the source code actually does reflect the test command.  At the end of the day, the code will indeed have the final say.

package com.acmesoft;

public class helloworld
{
  public helloworld()
  {
     System.out.println("hello world!");
  }

  public static void main (String[] args)
  {
     new helloworld();
  }
}

Hello world – template

In my opinion the most convenient feature of Maven is the ability to create an empty project.  There is a list of thousands of different types of template projects.  With a simple command it is possible to create the entire project from scratch – just add code.

mvn archetype:generate -DgroupId=de.acmesoft -DartifactId=helloworld -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false                                                                                                               
[INFO] Scanning for projects...                                                                                                       
[INFO]                                                                                                                                
[INFO] ------------------------------------------------------------------------                                                       
[INFO] Building Maven Stub Project (No POM) 1                                                                                         
[INFO] ------------------------------------------------------------------------                                                       
[INFO]                                                                                                                                
[INFO] &gt;&gt;&gt; maven-archetype-plugin:2.4:generate (default-cli) @ standalone-pom &gt;&gt;&gt;                                                     
[INFO]                                                                                                                                
[INFO] &lt;&lt;&lt; maven-archetype-plugin:2.4:generate (default-cli) @ standalone-pom &lt;&lt;&lt;                                                     
[INFO]                                                                                                                                
[INFO] --- maven-archetype-plugin:2.4:generate (default-cli) @ standalone-pom ---                                                     
[INFO] Generating project in Batch mode                                                                                               
[INFO] ----------------------------------------------------------------------------                                                   
[INFO] Using following parameters for creating project from Old (1.x) Archetype: maven-archetype-quickstart:1.0                       
[INFO] ----------------------------------------------------------------------------                                                   
[INFO] Parameter: basedir, Value: /tmp                                                                                                
[INFO] Parameter: package, Value: de.acmesoft                                                                                         
[INFO] Parameter: groupId, Value: de.acmesoft                                                                                         
[INFO] Parameter: artifactId, Value: helloworld
[INFO] Parameter: packageName, Value: de.acmesoft
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] project created from Old (1.x) Archetype in dir: /tmp/helloworld
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.132s
[INFO] Finished at: Sun Dec 11 23:46:15 CET 2016
[INFO] Final Memory: 13M/223M
[INFO] ------------------------------------------------------------------------

The generate command not only creates the directory structure but also creates the pom.xml file.  You can see how the command line parameters end up in the appropriate location in the pom file, which is also created automatically.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.acmesoft.secondexample</groupId>
  <artifactId>secondexample</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>secondexample</name>
  <url>http://maven.apache.org</url>
  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency> 
  </dependencies>
</project>

This pom file is actually also a very basic file.  The very minimum is the same as the manually created pom file in the manual section.    Maven automatically added Junit in order to allow the unit tests that it also created to compile.

package de.acmesoft;

import junit.framework.Test;
import junit.framework.TestCase;
import junit.framework.TestSuite;

/**
 * Unit test for simple App.
 */
public class AppTest 
    extends TestCase
{
    /**
     * Create the test case
     *
     * @param testName name of the test case
     */
    public AppTest( String testName )
    {
        super( testName );
    }

    /**
     * @return the suite of tests being tested
     */
    public static Test suite()
    {
        return new TestSuite( AppTest.class );
    }

    /**
     * Rigourous Test 🙂
     */
    public void testApp()
    {
        assertTrue( true );
    }
}

The template also includes the source code for our hello world program as well.

package de.acmesoft;

/**
 * Hello world!
 *
 */
public class App 
{
    public static void main( String[] args )
    {
        System.out.println( "Hello World!" );
    }
}

One small difference between the manual project that I created is that using the template creates the package, the pom file and the directory structure all consistent with each other.

Hello world – log4j

The generic hello world prints the output to the standard output and doesn’t use external libraries for any task.  Some applications or utilities might be this simple but it is uncommon for any but the most trivial application to not use external libraries for some purpose.

To simulate such a case, I have modified the hello world application to use log4j for all output.

package com.acmesoft.secondexample;

import org.apache.log4j.PropertyConfigurator;                                                                      
import org.apache.log4j.Logger;                                                                                    

public class App                                                                                            
{                                                                                                                  
  public App()                                                                                              
  {                                                                                                                
     String cfgFileName = "log4j.properties";                                                                      
     PropertyConfigurator.configure(cfgFileName);                                                                  
     Logger logfile = Logger.getLogger(App.class.getName());                                                
                                                                                                                   
     System.out.println("hello world!");                                                                           
     logfile.info("hello world");                                                                                  
  }                                                                                                                
                                                                                                                   
  public static void main (String[] args)                                                                          
  {                                                                                                                
     new App();                                                                                             
  }                                                                                                                
}

Running hello world

My first test of the application failed for obvious reasons.

java -cp target/helloworld-1.0.0.jar com.acmesoft.secondexample.App

The reason was NoClassDefFoundError was thrown.  The application uses log4j so it will not work unless it is part of the classpath.

java -cp target/secondexample-1.0-SNAPSHOT.jar:lib/log4j-1.2.17.jar com.acmesoft.secondexample.App

Maven is just another way to compile an application, the Java basics don’t change.  Simply make sure that when running the program that all necessary dependencies are provided. However, Java programs cannot be compiled unless the code that it is dependent on are provided. This is done by adding dependencies to the pom file.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.acmesoft.secondexample</groupId>
  <artifactId>secondexample</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>secondexample</name>
  <url>http://maven.apache.org</url>
  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>log4j</groupId>
      <artifactId>log4j</artifactId>
      <version>2.0.1</version>
    </dependency> 
  </dependencies>
</project>

The dependencies block contains both a dependency for JUnit and Log4j.  The JUnit dependency was automatically added when the application was created by Maven (in order to support the automated testing) but I manually added the Log4j block as it would be necessary for the program.

Hello world – log4j revised

The good news is that Maven can be configured to include the dependencies as part of the “package”.  This way it is possible to create the equivalent of a statically linked application that can simply be copied from directory to directory and it will still work.

The list of dependencies don’t change but what does change is the new “build” block that embeds the dependencies into the resulting jar.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.acmesoft.firstexample</groupId>
  <artifactId>firstexample</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>firstexample</name>
  <url>http://maven.apache.org</url>

  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>log4j</groupId>
      <artifactId>log4j</artifactId>
      <version>1.2.17</version>
    </dependency> 
  </dependencies>

  <build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>2.1</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <transformers>
                            <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <mainClass>com.acmesoft.firstexample.App</mainClass>
                            </transformer>
                        </transformers>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

</project>

Once the dependencies are embedded inside the jar file, you can simply run the application without worrying about the rest of the dependencies.

java -cp target/firstexample-1.0-SNAPSHOT.jar com.acmesoft.secondexample.App
Posted in programming | Tagged , | Leave a comment

creation of a more responsible padlock

I have been cleaning up my flat in preparation for the move when I ran across a few small TSA locks.  Sure I have used them when traveling to and from the United States but it certainly seemed to be a pretty sketchy to keep your things safe.

Oh, sure it must have seemed like a good idea when it first came up in that meeting.

Why don’t we require all the passengers use locks that we have keys to.  It would make it a lot easier if we need to search a suitcase that looks questionable.

I suspect it might indeed make things faster if there is a key-chain on the wall with the master keys for all of the locks.  I couldn’t find any statistics on how many suitcases are opened every year but I suspect that despite the number opened the percentage is pretty small compared to the 432 million scanned.

It does actually make a lot of sense having these master keys as some traveler might be trying to smuggle money, drugs or art artifacts in their suitcase.  These activities are all illegal and most law abiding people would agree that the law should be obeyed the criminals should be caught and punished. Right?

The only problem with that is that scenario is that people are only human and some of the species are drawn to taking short cuts in order to achieve their goals.  Some people might even be using their position to get copy of such a key.  The security on these keys is (hopefully) very tight but it is not necessary enough to prevent someone from stealing a key.

People can be greedy or unscrupulous but they also have another attribute – sometimes they are sloppy.  Just like accidentally saying to much to a boss or a friend, during an article on TSA travel security, a picture was published of these master keys.

I am not a locksmith but it apparently just a hop skip and a jump from a photograph of a key to creating a physical key of your own.

So while it might seem like a good idea at first to have a number of different TSA padlocks and have the authorities control the keys it really isn’t. Humans find it is pretty much impossible to keep a secret and have a very hard time with good operational security.

Despite having a limit number of locations where these locks will be used there is a market for making bootleg keys available – even years after the key leaks.  Possible to purchase keys for these master locks on ebay.

I am not really interested in the TSA locks.  I am giving serious consideration to simply purchasing any old lock and using it.  There will be two outcomes.  The first is that my suitcase will not be selected and thus the lock doesn’t make a difference.  The second is that my suitcase is selected and they discover that I am using a non-TSA lock.  They will just cut it off and go about their business.

Both of these options would make me more secure than having a lock that virtually anyone could open (pilfer things from my suitcase) and then lock it up like nothing ever happened.

Not physical locks but Encryption

The record with physical security isn’t so rosy but it seems considerably more secure than in the electronic world.  It seems like not a month goes by without some sort of huge data breach being published.

This isn’t talking about using “responsible encryption“, this is simply talking about how poor the current state of electronic security is.

These breaches of occurred just because the “bad” guys go where the data is.  This level of data protection is not acceptable but unfortunately this seems to be the norm.

This is the track record of companies whose livelihood depends on staying in business based on their appropriate control of customer data.

With this track record, does it seem like a good idea to hand over the keys for everyone’s electronic key to the government?  Well, it is the government, perhaps they are better at keeping control of their data.

It is possible to attempt to change the behavior of people or countries by changing the laws. Right?

Responsible Encryption

It has been suggested, yet again, that there is some possible way to both have safe encryption and yet allow the government easy access to it whenever they want.

The other problem is that humans are a really tricky lot and can come up with a number of clever solutions other than normal encryption.

It is not clear if the US president does intend to pressure companies to use breakable encryption or encryption with backdoors but it would probably have unexpected consequences.

The harder naughty people are chased, the further from the common methods of communication they will stray.  If phone encryption is no longer safe, they will brew their own.  If that is not possible some of these other historic encryption methods may be used in the dark corners of the internet.

Responsible encryption is strong encryption.  In my opinion, it is criminal to weaken encryption that will inevitably end up not only in a social app, banking app or perhaps password keeper.  Securing peoples personal information in this age of no privacy is being safe and responsible.

Finally I did see a response to this article that pretty much sums up the hypocrisy of a Republican president wanting more control.

Beyond all the absurdity of this all is the fact that the party that’s been yelling about “limited government” for decades is now suggesting a central government entity collect the passcodes to everyone’s data “just in case”.

– Ihave8eggs

In the United States it is not uncommon to see a bumper sticker.

When guns are outlawed only outlaws will have guns

This may be true for something physical like weapons but it is even more true for encryption.  The genie has been let out of the bottle decades ago and it is not possible to put the genie back into the bottle.  This is because strong encryption is no longer only in the realm of governments.  It is possible to download open source solutions that are just as tough to crack as something the NSA uses.

This is the problem that law enforcement faces.  Your dirty little secrets on your smart phone are probably as secure as the data stored at Langley.

Posted in Soapbox | Tagged , , , | Leave a comment

Maven the alternative build tool

Why maven

Maven is a tool for building software. Doesn’t there already exist a number of tools for doing this exact task?  Two different tools that do this are the classic make utility or the much newer Apache Ant.

The make utility requires a script that defines all of the objects and their dependencies. Make forces you to define every object and their dependencies.  That is a bit of a harsh analysis of the make tool as it is possible to create more generic rules for compiling objects from source but in general this is accurate description of make.

Apache Ant is a few more steps along the automation path from make.  You define some of the targets and a few rules in your build.xml file for compiling code, making jar files or other general commands.  There is no default defined directory structure for the source files or external libraries.  You need to define the structure each time you create a new program.

Maven has a number of differences from other build tools.  There is a very specific directory structure for each of the types of programs or objects that Maven can build.  In addition to the actual directory structure Maven has the concept of a the software lifecycle for the objects it builds.

Maven will perform all the steps in the lifecycle from the start up until the phase that you select.

  • validate – perform a check to ensure that all the necessary files are available.
  • compile – compile the source code of the project.
  • test – runs the unit tests that have been defined.
  • package – create the final package for the code.  This might be a JAR, WAR or EAR file.
  • install – install the package into the local repository.
  • deploy – install the package into the remote repository.

Thus the program is compiled before it is tested and tested before it is packaged.

It is possible to setup a similar flow using Apache Ant but then you are responsible to create lifecycle from scratch.  This includes adding additional logic for doing testing and implementing logic for installing the compiled objects.

In addition to all the other built-in logic that Maven contains it also is well aware of the different types of output objects such as jar or war files.

Maven Repositories

Another major difference between these other tools is that Maven has its own repository of jar files and other artifacts that might be needed when building objects.  Maven can store multiple versions of the various artifacts in its repository.

The central repository is full of all sorts of open source jar files so it is quite possible that if your project uses one of these jars (ie log4j) then it will already be available.  You simply add your required dependency and it will be downloaded to your computer and used on the next build.

Each object in the repository is defined by the unique combination of three different keys.

  • groupId
  • artifactId
  • version

The combination of these three keys creates a unique value that refers to the object specifically.  This built in repository concept actually really simplifies dealing with libraries if you only need open source libs.

Of course it is also possible to add other third party libraries.  Using libraries that are not already in the repository is a bit different and will be described later.

Projects and project structures

One of the advantages to Maven is the structure that it provides.  Maven contains a lot of templates and standard directory structures.  There is a structure available if you simply need to create a java jar file but there is also a template if you want to create restful web applications.

Of course, it is possible to create these directory structures manually but another feature of Maven is that you simply pick which type of project you wish and it will create the entire directory structure for you.

Testing your work

It goes without saying that providing unit tests would be both a best practice and is easy enough to do if you start at the beginning.  Performing the unit tests is part of the lifecycle and because they are run before creating any packages it should help to guarantee a higher quality of program.

Summary

Maven is a enterprise quality build tool that supports more than just the generation of programs or libraries.

Maven is big but I will attempt to describe some of the basics for anyone who wishes to expand their tool set to include it.

 

Posted in programming | Tagged , | Leave a comment

installing maven on linux

Maven offers quite a bit as a enterprise build tool.  It can help ensure that standards are followed and offers support for automated testing.  This tool is actually really simple to install.  It is just a matter of unpacking it and adding it to your path.

This is described on a lot of locations on the internet.

https://intojava.wordpress.com/2016/05/05/install-maven-linux-mint/

The nice thing about most of the major linux distributions is that most major packages provide a package to install Maven.  This takes care of both the installation and setup of the path.

However, in my test this also installed a lot of other java packages that should not have been necessary.

apt-get install maven
Posted in Command line, Setup From Scratch | Tagged | Leave a comment

command line fun – finding mistakes

I knew that I had mistyped but it was too late.  I tried to exit out of vi in a hurry and accidentally exited but saved the file under a new name.  This wouldn’t have been so bad if the new name didn’t consist of a single control character.

My fears were confirmed when I did a directory listing.  Not only did it make an annoying sound but you could see in the file listing that the files were no longer lined up.

This isn’t the first time that my fingers got the better of me, so I simply ran the ls command and asked that it display the inode number of each of the files.  Normally that would have been fine but my control character, perhaps ^H, was actually removing the previous character displayed.

This means that my inode number was any one of ten possible values.  There are other ways to try and determine exactly what inode number a file is but before trying that I simply did a long listing using ls.

This actually did show that my inode number was 540.  I am actually a bit surprised that you cannot use this information directly to delete the file with the rm command (on Solaris, also on Linux?)

This information however can be used in conjunction with both the find and rm commands to delete this tiny mistake.

Instead of the familiar -name option for the find command there is a somewhat less used   -inum option.  As this option name implies, it will look only for files that have this inode number.

Once it is possible to find the actual file it is a simple matter of using the -exec option of find to remove the file.

find . -inum 540 -exec rm {} \;

Extra credit method

There is another way to find an inode number using just variations of the find command.

find . -name “*”  -ls

This will just do a directory listing of the files under the current working directory and run a similar directory listing at the same time.  Deleting the file is done using the same find and remove as described above.

Posted in programming | Tagged , | Leave a comment

Got a new tablet for the price of fastfood meal …

… well, that is a bit of an exaggeration.  I owned a Samsung Galaxy S tablet which no longer worked.  It had the darndest symptoms.

  • doesn’t charge past 77 %
  • takes a day to get to 77%
  • spontaneously restarts shortly after startup

The plan was to take the tablet apart and try and repair it (despite not being a certified hardware repair person)  After all, just how hard could it be?

The repair

The tablet wasn’t that old and was in otherwise perfect condition.  The Internet implied that there were a couple of different parts that might solve the problem.  I was hoping that simply replacing the usb port would be the problem.

https://www.amazon.de/gp/product/B01MFH19ZY/ref=oh_aui_detailpage_o02_s00?ie=UTF8&psc=1

It would have been perfect, the cost for a new port is less than 6 Euros.  There are a number of really good youtube videos to help for replacing the usb port  or how to replace the battery.

The hardest task was actually getting the back of the tablet off.  The video for replacing the usb port makes it look pretty easy.  In that video the back itself is simply pulled off and you don’t need any special tools for that.

It might be obvious from this picture but when I removed the back I actually pulled the screen out from the other side. This is actually quite similar to how it is removed in the battery video.

Removing the charger port is actually trivial, especially if you have removed the screen in the way that I did.  The reason is that the back of the tablet is actually two parts.  The first is a rim that goes around the tablet and holds both the screen in from one side and the back from the other side.  This rim actually would make it a bit more difficult to get the port off.

Near the usb connector port is a two screws that hold the port to innards.  Once the screws are removed a small bit of pressure and the port and the sdcard hold pop right off except for a small connector that also attaches to the board.  The other end is a flat cable.  The connector is slightly different as it flips up rather than slides to release the cable.

I had hoped that this six euro part would fix the problem but as it turned out this did not solve my problems. The new port did appear to work but once I removed the power cord from the tablet the device would barely make it to the startup screen.

https://www.amazon.de/gp/product/B00R1A0VK6/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1

The battery is actually even easier to remove than the charging port.  There are four screws that held it in place and then the power cord snaps onto the board.

Lessons learned

Dealing with the insides of this particular tablet was actually very easy to work with.  The most difficulties that I had was trying to get the cover off.  I think that I must have been looking at a different video and it didn’t make the task easier.

I actually had a small set of tools that offered a number of possibilities but I used the small triangular pic.

The correct tool appears to have been this tool that I happened to overlook.

Everything works

For the low low cost of about 38 Euros I managed to get my tablet to work.  I wouldn’t have done tried to take this apart if it hadn’t been non-functional and out of warranty. I don’t know how the repairable the newer Samsung tablets are but the older ones appear to be very repairable.

Posted in Setup From Scratch | Tagged | Comments Off on Got a new tablet for the price of fastfood meal …

command line fun – temporary data

Most of the scripts that I need to write don’t actually need to manipulate a lot of data.  It is usually enough to pipe that data through a command or two and use the result.  When this isn’t acceptable then it is usually a small enough amount of data that I can save it in a temporary file with the same name.

Yet there might be a number of cases where truly unique names are necessary.  One example might be if the script gets run many times in parallel but still needs to store some temporary data.

There are a few different methods of generating a unique name.

Process Id

Each Unix process has a unique id and that can be used as part of the file name.  There are some special variables that can be accessed and one of them is $$.  This bash shell variable will be the actual process id of the current shell.

TEMPFILE=mytempdata.$$

This actually should be pretty safe for a lot of situations depending on the volume.

Timestamp

Another fairly classic method of creating a unique name would be to create a timestamp.  This is a terrible device for coming up with a unambiguous name unless the script in question is being run in serial.

TEMPFILE=mytempdata.`date '+%Y%m%d-%H%M%S'`

This example simply uses the date command to display the current date and time.  This date and time is redirected using the backtick.  This can also be done using the more modern method of redirecting the output.

TEMPFILE=mytempdata.$(date '+%Y%m%d-%H%M%S')

Operating system support

There exists the mktemp command which will create an absolutely unique name based on the names of the files or directories in the destination directory.  There is one “little” side effect of this command and that is that the temporary file or directory is created. The file is a zero length file while the directory is a normal directory.

Using the same technique as in the previous example it is possible to both create a unique name while also capturing the name in a variable.

TEMPFILE=$(mktemp /tmp/mytempdata.XXXX)

TEMPDIR=$(mktemp -d /tmp/mytempdata.XXXX)

The operating system cannot know how many free files you might need in your directory. The mktemp command will replace the capital letter X with the unique alphanumeric value.  If more than 999 files might be in use at a time then use four X’s.  The exact location of the capital X is not important.  It can be set as an extension or just a value within the name.

 

Posted in programming | Tagged , | Comments Off on command line fun – temporary data

virtualbox disk too small ?

The first few times I used VirtualBox it wasn’t clear why you would ever want to create a virtual disk that is dynamic.  I could only envision having three machines each with a disk that wasn’t fully expanded only to run out of real storage when I really needed it.

However, just like real machines it is possible to full up the hard disk and want just a tiny bit more space.  Of course you can add a second virtual disk to your machine but there is another option.  It is actually possible to add to a small extra bit of space to your virtual hard disk, but only under once situation.  It is not possible to expand a fixed size disk beyond its initial size but it is possible to extend a dynamic disk to a larger size.

The steps to extend a virtual harddisk is actually pretty simple.

  1. extend the virtual disk image (vdi)
  2. mount gparted inside the virtual machine
  3. select and extend the partition

Original state

The disk drive associated with my virtual machine was only 10GB.  This is actually more than enough for small tests.

Extend the virtual disk image

This first step needs to be performed on a disk image that is not currently being used by virtualbox.  This step is done by running the command line program vboxmanage.  Simply give the total size of the disk image and the disk is extended.

The command both gives feedback while running and runs really fast.

vboxmanage modifyhd <qualified path to disk> --resize <# of megabytes>

Virtualbox can display the disk size as soon as the command has finished running.

Mount Gparted inside the VM

Simply resizing the virtual disk image is not actually enough.  This will make the disk itself larger but it will not make any more space available to the client operating system inside of the virtual machine.  This space can be allocated to the partition by using some tool to extend the partition.

Gparted is one of the easiest tools to use to enlarge the partition.  However, just like many other Linux tools it needs to be run on a volume that is not currently mounted. In order to do that simply download the gparted live disk and mount it to the virtual machine.

The Gparted live disk will boot up just like on any physical machine, it will bring up a grub menu and let you start the Gparted live installation.

Actual extension of file system

The actual extensino of the file system in this particular case is trivial.  Simply select /dev/sda5 and resize this to use the extra two gigabytes that have been added.

This particular case was trivial by design.  I didn’t want to move the swap partition system to the end of the disk and move the free space to the end of the partition.

You can see that the space has been allocated to the partition /dev/sda2 but not yet being used by the file system.  The next step is to expand the actual filesystem on /dev/sda5

It may seem like a lot of effort to simply add a few gigabytes or more space to a virtual disk image but the entire process can be done in a few minutes.

 

 

Posted in programming | Tagged | Comments Off on virtualbox disk too small ?

Virtual hardware -better than the real thing

One of the really neat things about the here and now is all those spare CPU cycles that are available.  With enough CPU power it is possible to create special effects such as adding dinosaurs to movies, bringing historical figures into contemporary films or perhaps bringing space ships and aliens to the big screen.

One of the non-entertainment applications that can take advantage of all this extra power is virtual machines or emulators.

Term Description
Virtual machine or VM Simulating parts of the hardware while providing a virtualized access to the rest of the real hardware.  It is important that the software being virualized has the same architecture.
Emulation The entire hardware is simulated without relying on the CPU to run

All of this sounds good, but what practical things can we use the virtual machine for?

  • Running old programs
  • Safe web surfing
  • Test environments
  • Run an alternate operating systems
  • Create a personal cloud
  • Reuse old hardware

The good news is that there are quite a few free and commercial options available to create such virtual machines. There are a number of companies that provide their own solutions ranging from free through paid versions.  VMWare has their full fledged hypervisor VMWare workstation as well as a lessor version for free VMWare player.  The differences between these two have blurred over the years since the free player was released.

One of the other large companies that also has a collection of free and paid hypervisors is Oracle.  Their product VirtualBox is freely available and like VMWare runs on either Linux or Windows.

Microsoft is also has its own hypervisor which is called virtual pc.  The difference between their solution and the previous two is that the previous two support both Linux and Windows while the Microsoft solution can also virtualize MS Windows.

In the past I used to use VMWare workstation but over time I have come to appreciate VirtualBox.  It is actually quite intuitive to use and it also has a number of different networking options.

VirtualBox Installation

I downloaded my package from Oracle and attempted to install VirtualBox.

# dpkg -i virtualbox-5.0_5.0.10-104061-Debian-jessie_amd64.deb

There are a number of pages to help assist in the installation of VirtualBox on Linux so I won’t be going into it here.

Using VirtualBox

I have been using VirtualBox so much the last few days to try out a number of different Linux distributions.  It is amazing how little configuration is really needed in order to Just download your favorite Linux distribution and create a new VM.

Yet, I wanted to play an old windows game so I thought I would try to setup an windows 7 environment.  The process is actually no more complicated.  Below is a few screenshots of the setup of the virtual machine.

Once this is done the process is virtually identical to using a proper personal computer.  Just put a dvd into the drive and install like normal.

Give a hint to virtualbox which operating system it has to support.

Setup drag-n-drop or clipboard support

 

Setup the amount of memory for the VM

Assign the amount of cpu/threads to our machine.

Setup VirtualBox to use hardware support (if available).

 

Select more hardware support

 

Define the size and location of the virtual drive.

 

Setup the network adapter – maximum of 4.

As previously mentioned, the process is no different than with a proper physical computer but there is one little thing that must be done in order to connect the host machine’s dvd.

Simply select your dvd (or cdrom) drive

Once all of this is setup simply select your virtual machine setup from the list and press the green arrow to start the machine.  The disk or ISO image will boot up and you can proceed with the rest of the operating system installation as normal.

How good is VirtualBox

I was curious just how good VirtualBox is.  I have installed a lot of Linux VM’s with no discernible problems nor did I have any problems installing Windows 7 for this experiment.

The most honest test that I could think of that would really put VirtualBox to the test would be to try and install a “restore dvd” from an old laptop.  I was actually pretty amazed that the disk booted up just fine … to a point.

The restore disk discovered that it was not running on an HP laptop.  The problem was that the restore disk was created so it would work on an HP, well that particular model HP actually.

This bit of trickery is done by the use a ACPI table (SLIC) in the bios.  It is an interesting history of how Microsoft has used key mechanisms to enable their operating systems.  I hope to cover that one soon.

Posted in Setup From Scratch | Tagged | Comments Off on Virtual hardware -better than the real thing

airplane data security

When I think about airlines I think about a lot of things but I don’t think of high tech. That isn’t very realistic as jet engines are pretty high tech devices and the ability to book reservations over the internet is high tech – simply search by date and city and presto you can select (and pay for) flight within minutes.

It is a pretty clever but all of this “internet access” is just a thin layer over the real backbone of the airlines.  The airlines and travel agencies have had their own legacy system in place to allow for booking of flights.  This system, a so called global distribution system, created to coordinate between the airlines and the travel agencies to prevent selling a seat twice.

However, the older the system the less likely it is to have been programmed to withstand a concentrated external attack by a dedicated attacker.

The neat thing about the internet age is not only the various technologies that exist but also the various white hat hacker groups that exist to investigate these technologies and cast light on these important issues.  One of the oldest such groups involved in this is the chaos computer club which has existed since the early eighties.

The chaos computer club does some amazing clarifications of the weaknesses that they investigated.  A few months ago, they had a Karsten Nohl give a speech about the security or lack of security of this very important system for anyone who flys.

This speech is both an amazing speech on the clarity of how the system works but terrifying how poorly such an important system is protected.

One of the most worrisome parts of this speech, especially for privacy advocates, is where Karsten points out that there is no access logs for this system.  (at 44:20)  This means that either a person or government can track a person and it would be impossible to know that someone has been checking out your itinerary.

There are a lot of things to take away from this speech but one of the surprising for most people is that the luggage tags and boarding passes need to be securely disposed of.

Posted in Soapbox | Tagged , , | Comments Off on airplane data security