command line fun – unlimited command line length

While looking into another problem, plethora of shells, I was initially afraid that some of my fooling around was causing some odd side effects.  While investigating I ended up with a different solution that was actually overkill but it help eliminate other options as a source of the problem.

My solution was to create a file with the names of all of the files that I needed to process and use that as my input.   This allowed me to look at as many files as I wanted to without seeing anything like this.

-bash: /usr/bin/ls: Arg list too long

This isn’t the only solution, nor even the most efficient but it sure is a fun one that uses both head and tail on the same line.  A more efficient solution would probably to use xargs, find or perhaps a more efficient shell script.

Rather than to do the efficient, I am going to show off the fun solution.  This is something you could use even in a production environment as long as the number of files is reasonably small.

I decided to use both head and tail to pick out lines from a file for processing.  Simply save a list of all files to a temporary file and then process each file one at a time.

The fun example

INPUT=/home/bob/input
TMPFILE=`mktemp /tmp/XXXXXX`

pushd `pwd`
cd $INPUT
ls -1 > $TMPFILE
CNT=`cat $TMPFILE | wc -l`
IDX=1
while [ $IDX -le $CNT ]
do
   NAME=`head -n ${IDX} $TMPFILE | tail -1`

   echo processing $NAME
   # do some processing here

   IDX=$(($IDX +1)) 
done
popd
rm $TMPFILE

The actual interesting part is where the variable NAME is being assigned but before getting ahead of myself, here is a blow by blow explanation of the script.

# Line Description
1. INPUT=/home/bob/input set up our input location
2. TMPFILE=`mktemp /tmp/XXXXXX` create a unique filename in the /tmp directory
3. pushd `pwd` save our current directory
4. cd $INPUT change to input directory
5. ls -1 > $TMPFILE save a list of file names (no directory) into our temporary file.
6. CNT=`cat $TMPFILE | wc -l` get a count of the number of lines in our file.
7. IDX=1 which line of the file we are on at the moment.
8. while [ $IDX -le $CNT ] while there are lines left to process
9. do
10. NAME=`head -n ${IDX} $TMPFILE | tail -1` get the entry from the file.
11. echo processing $NAME in this case echo the filename to the screen.
12. # do some processing here nothing to see here, move along.
13. IDX=$(($IDX +1)) increase index to the next line
14. done
15. popd change back to original directory
16. rm $TMPFILE remove temporary file.

The code is pretty obvious what it is doing for most of the script.  The interesting lines might be 2 and 10.

The mktemp command simply creates a unique file that is six characters long designated by the XXXXXX format.  This is really useful as the operating system will do the hard work for you.  It is even possible to create a subdirectory with a file or just a unique subdirectory.

Line number ten is actually pretty simple when you think about it. The head command will take that many lines from our temporary file.  the first time through this will be the first line.  the tail command will then take the last line from this batch.  When the index is 1 both the head and tail will deliver a single line.

When the index is larger (ie 5) then more lines will be delivered.  The head command will deliver all of the previously processed lines along with one new one.  The tail command will pull off that one new line so it can be processed.

This method of processing data might also be interesting if there are spaces in the filenames or paths.  You would know with certainty that each line represents a single parameter or value.

Again, this is a terribly inefficient way of processing through a million lines, especially if these lines are each hundreds of characters long.

Posted in Command line | Tagged , | Comments Off on command line fun – unlimited command line length

Gentlemen’s vandal

We were watching the BBC world news a few days back and it must have been a slow news day. The reporter was speaking with someone whose face was hidden. It turned out that this particular vigilante defaced buildings and has been doing so for the past 13 years.

No, he wasn’t putting up gang graffiti letters nor faces, animals or general art. His changes were very very tiny. So tiny that the actual shop owners in some cases didn’t recognize anything was different. So what was the change?

He would either add or remove apostrophe’s as required.

Ie.

Amys Nail’s -> Amys Nails

The interviewee never said what his profession is but it could easily be that of an English teacher considering how my teacher was during English class.

It is hard to encourage vandalism but this particular type is at least a refreshing change from the norm.

A “grammar vigilante” sneaks around at night fixing an infuriatingly common error on public signs

Posted in Soapbox | Comments Off on Gentlemen’s vandal

Restful services in Java – client and server

My previous restful client example and server example were perfect, well, not exactly.  They worked but they pretty much assuming a perfect world.  That is to say that there is never a communication error or a server error.

This small example will return one of the magazine objects.  Depending how you call it, it will return either the object directly or the xml that you can marshal into the object itself.

// example 2/3 setup
ClientConfig config1 = new ClientConfig();
Client client1       = ClientBuilder.newClient(config1);
WebTarget target1    = client1.target(uribase);
		
// example 3
magazine xmlobj = target1.path("/Communicate/hello").request().accept(MediaType.APPLICATION_XML).get(magazine.class);
showMagazine("pure xml object", xmlobj);

However, it should be pretty obvious in the previous simple server code that if for any reason the server cannot find the data it cannot really communicate that fact back to the client.  Well, it cannot if you are returning magazine objects or general strings.

New and better server

The solution is to return a response record.  This record will contain both the error code as well as our object.  The code is actually not too different from from the previous example but there are some major differences.

The first is that we are no longer returning either xml(string) nor magazine objects but rather we are returning Response objects.    This allows us to set the status to one of the values that matches the result of the call.

Response.Status.OK

It is possible to use one of the predefined values such as Response.Status.OK but it is just as acceptable to put the actual value of 200.

Here is a list of the values that are defined for us to use.

Response codes Description
OK 200
CREATED 201
ACCEPTED 202
NO_CONTENT 204
MOVED_PERMANENTLY 301
SEE_OTHER 303
NOT_MODIFIED 304
TEMPORARY_REDIRECT 307
BAD_REQUEST 400
UNAUTHORIZED 401
FORBIDDEN 403
NOT_FOUND 404
NOT_ACCEPTABLE 406
CONFLICT 409
GONE 410
PRECONDITION_FAILED 412
UNSUPPORTED_MEDIA_TYPE 415
INTERNAL_SERVER_ERROR 500
SERVICE_UNAVAILABLE 503

The code in this “server code” is not much larger than the original code.

Another one of the changes is that the URI is returned as part of the response object.  This URI should point to the the data in the server.  This is especially interesting if a new object is created on your server.  The server then will return the URI to reference that object.

The final difference is that we pass back our object as part of the Response object.  In the below code this is done as a simple string but it could just as easily be a your own Java object.

package com.acmesoft;

import java.net.URI;
import java.net.URISyntaxException;

import javax.ws.rs.Consumes;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.UriInfo;

@Path("/Communicate")
public class HelloWorldEx {

	/*  
	 * http://192.168.178.70:8080/SecondRestful/Communicate/mag
	 */	@GET
	@Path("/mag")
	@Produces( {MediaType.APPLICATION_XML})
	public Response sayHelloXml(@Context UriInfo ui) throws URISyntaxException
	{
		String retval =  "" +
				"2016" +
				"6" +
				"Economist" +
				"9.99" +
				"";
		String path = ui.getAbsolutePath().toString() ;
		URI uri = new URI(path);

		return Response.created(uri).contentLocation(uri).status(Response.Status.OK).entity(retval).build();
	}

	/*  
	 * http://192.168.178.70:8080/SecondRestful/Communicate/mag
	 */
	@GET
	@Path("/mag")
	@Produces( {MediaType.APPLICATION_JSON})
	public Response sayHelloJSON(@Context UriInfo ui) throws URISyntaxException
	{
		String retval =  "{ \"magazine\": {\"year\":2016,\"month\":6,\"title\":\"Economist\",\"price\":9.99} }";
		
		String path = ui.getAbsolutePath().toString() ;
		URI uri = new URI(path);

		return Response.created(uri).contentLocation(uri).status(Response.Status.OK).entity(retval).build();
	}

	/*  
	 * http://192.168.178.70:8080/SecondRestful/Communicate/oucha_bunga
	 */
	@Path("{c}")
	@GET
	@Produces( {MediaType.APPLICATION_XML})
	public Response complexURI(@Context UriInfo ui, @PathParam("c") String msg) throws URISyntaxException
	{
		String mymsg = "Thinkin about " + msg;
		String xmlStr = "" + mymsg + "";

		// this is a really clever way to get the URI but
		// if you pass in a string, it will throw an error.
		// and it all comes down to the URI that is added to the
		// response object (below)
		String path = ui.getAbsolutePath().toString() ;
		URI uri = new URI(path);

		return Response.created(uri).contentLocation(uri).status(Response.Status.OK).entity(xmlStr).build();
	}
}

The final method in this server example is is not actually called by my client but it is a clever example of a URI that can accept anything, in this case a string, and then that string can be used in some clever manner.

New Client

The new client changes is actually quite similar to the new server changes.  As a matter of fact, the client changes are actually almost transparent. There is just a few small changes to accommodate the retrieving Response objects instead of strings or other java objects.

Once the Response object has been retrieved simply call the status method to find out the status returned from the server. The range of success codes is 200-299.

	if (response.getStatus() == 200)
	{
		xmlAnswer = response.readEntity(String.class);
		System.out.println(xmlAnswer);
		magazine converted = convertXmlToMag(xmlAnswer);
		showMagazine("xml text data", converted);			
	}
	response.close();

If the call was successful then simply call the readEntity method on the Response object.  Simply pass in the class of the object that you are trying to read out.  The object is retrieved just like magic.

In order to keep control on the system resources close the Response and the Client objects, failing to do so will cause your program to fail eventually due to resource issues.

package com.acmesoft;

import java.io.StringReader;
import java.net.URI;

import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.client.WebTarget;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.UriBuilder;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Unmarshaller;

import org.glassfish.jersey.client.ClientConfig;

public class TestClient 
{
	String uribase = "http://192.168.178.70:8080/SecondRestful";

	public void showMagazine(String desc, magazine value)
	{
		System.out.println("\n" + desc );
		System.out.println("year "  + value.getYear());
		System.out.println("month " + value.getMonth());
		System.out.println("price " + value.getPrice());
		System.out.println("title " + value.getTitle());
	}
	
	public magazine convertXmlToMag(String xmlstring)
	{
		magazine converted = null;
		JAXBContext jc = null;
		try {
			jc = JAXBContext.newInstance(magazine.class);
			Unmarshaller unmarshaller = jc.createUnmarshaller();
			StringReader reader = new StringReader(xmlstring);
			converted = (magazine) unmarshaller.unmarshal(reader);	
		}
		catch (JAXBException ex)
		{
			ex.printStackTrace();
		}
		return converted;
	}
	
	public WebTarget restConnect(String targetUrl)
	{
		URI uri = UriBuilder.fromUri(targetUrl).build();
		ClientConfig config = new ClientConfig();
		Client client = ClientBuilder.newClient(config);
		WebTarget target = client.target(uri);
		return target;
	}
	
	public TestClient()
	{
		ClientConfig config = new ClientConfig();
		Client client       = ClientBuilder.newClient(config);
		WebTarget target    = client.target(uribase);

		Response response  = target.path("/").path("Communicate").path("/mag").request().accept(MediaType.APPLICATION_XML).get(Response.class);
		System.out.println(response.getStatus());
		String xmlAnswer = "";
		if (response.getStatus() == 200)
		{
			xmlAnswer = response.readEntity(String.class);
			System.out.println(xmlAnswer);
			magazine converted = convertXmlToMag(xmlAnswer);
			showMagazine("xml text data", converted);			
		}
		response.close();

		response = target.path("/Communicate/mag").request().accept(MediaType.APPLICATION_XML).get(Response.class);
		if (response.getStatus() == 200)
		{
			magazine xmlobj = response.readEntity(magazine.class);
			showMagazine("pure xml object", xmlobj);
		}
		response.close();
		client.close();
	}
	
	public static void main(String[] args) 
	{
		new TestClient();
	}
}

The output is not terribly interesting, it is just a few print statements in order to prove that the client can retrieve the data.

 200
<Magazine><Year>2016</Year><Month>6</Month><Title>Economist</Title><Price>9.99</Price></Magazine>

xml text data
year 2016
month 6
price 9.99
title Economist

pure xml object
year 2016
month 6
price 9.99
title Economist

With the ability to verify the return status of the restful server calls it is possible to create much more interesting services than displayed here.

I have attempted to create more than your typical “foo” “bar” type example program. My final blog on restful services will be a fairly complete service that could be used to provide price data supplied by several “banks” on various exchanges.

The data is actually pseudo randomly generated and retrieved and displayed by web browser using java script to fetch, extract and display the data from a JSON formatted string.

Posted in programming, Setup From Scratch | Tagged , , | Comments Off on Restful services in Java – client and server

Restful services in Java – simple client example

I suppose that a restful server isn’t all that useful unless there is a way to query this information back and use it in some useful way.

It is easy enough to test this out the server from the command line using the curl command.  Simply select the return type and select the URI.

curl -H “Accept: text/plain” http://192.168.178.70:8080/FirstRestful/Communicate/hello

 

curl -H “Accept: application/xml” http://192.168.178.70:8080/FirstRestful/Communicate/hello

This is the best starting point as we can pretty much guarantee that the server is working. Should any problems occur during the development of the client we can pretty much point to the client as the culprit.

Java client

My first web services client will be essentially just a plain old java program.  Although I am using java for my client I can easily imagine that there are some systems that were programmed in some older languages that use also use restful services in this same manner.  Not only communicate with a web server but to provide an easy way to enhance those systems with abilities that were didn’t exist when they were created (I am looking at you COBOL ).

Jersey makes the process of making restful calls easy from Java.  The restConnect method groups together the necessary logic for accessing a URI that is passed in.

Once these steps are done the object is a WebTarget is initialized and it is possible to call the restful service.

In this example client example I use use three slightly different methods for setting up the URI.  The two more flexible calls (lines 68 – 78) setup a base URI and then adds to the path.  This might be the best way if quite a few similar calls were necessary.

When doing the restful calls you also pass in the datatype that you wish to be returned back.  This then does the proper call to the server to get the correct return type.

/**
 * 
 */
package com.acmesoft;

import java.io.StringReader;
import java.net.URI;

import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.client.WebTarget;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.UriBuilder;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Unmarshaller;

import org.glassfish.jersey.client.ClientConfig;

public class TestClient 
{
	String uribase = "http://192.168.178.70:8080/FirstRestful";
	String urifull = uribase + "/Communicate/hello";

	public void showMagazine(String desc, magazine value)
	{
		System.out.println("\n" + desc );
		System.out.println("year "  + value.getYear());
		System.out.println("month " + value.getMonth());
		System.out.println("price " + value.getPrice());
		System.out.println("title " + value.getTitle());
	}
	
	public magazine convertXmlToMag(String xmlstring)
	{
		magazine converted = null;
		JAXBContext jc = null;
		try {
			jc = JAXBContext.newInstance(magazine.class);
			Unmarshaller unmarshaller = jc.createUnmarshaller();
			StringReader reader = new StringReader(xmlstring);
			converted = (magazine) unmarshaller.unmarshal(reader);	
		}
		catch (JAXBException ex)
		{
			ex.printStackTrace();
		}
		return converted;
	}
	
	public WebTarget restConnect(String targetUrl)
	{
		URI uri = UriBuilder.fromUri(targetUrl).build();
		ClientConfig config = new ClientConfig();
		Client client = ClientBuilder.newClient(config);
		WebTarget target = client.target(uri);
		return target;
	}
	
	public TestClient()
	{
		// example 1
		WebTarget target = restConnect(urifull);
		String plainAnswer = target.request().accept(MediaType.TEXT_PLAIN).get(String.class);
		System.out.println("raw (text)data");
		System.out.println(plainAnswer);
		System.out.println("");
		
		// example 2/3 setup
		ClientConfig config1 = new ClientConfig();
		Client client1       = ClientBuilder.newClient(config1);
		WebTarget target1    = client1.target(uribase);

		// example 2
		String xmlAnswer = target1.path("/").path("Communicate").path("/hello").request().accept(MediaType.APPLICATION_XML).get(String.class);
		magazine converted = convertXmlToMag(xmlAnswer);
		System.out.println(xmlAnswer);
		showMagazine("xml text data", converted);
		
		// example 3
		magazine xmlobj = target1.path("/Communicate/hello").request().accept(MediaType.APPLICATION_XML).get(magazine.class);
		showMagazine("pure xml object", xmlobj);
	}
	
	public static void main(String[] args) 
	{
		new TestClient();
	}
}

Below is the magazine class with the appropriate markups to easily do the JAXB conversion between plain old java objects and XML.

package com.acmesoft;

import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlRootElement;
import javax.xml.bind.annotation.XmlAccessType;

@XmlRootElement(name="Magazine")
@XmlAccessorType(XmlAccessType.FIELD)
public class magazine {

	// Microsoft Exchange values 
	@XmlElement(name="Year")
	private  int year; 

	@XmlElement(name="Month")
	private  int month; 

	@XmlElement(name="Title")
	private   String title; 

	@XmlElement(name="Price")
	private   double price;

	public magazine()
	{
	}
	public magazine(int year, int month, String title, double price)
	{
		this.year = year;
		this.month = month;
		this.title = title;
		this.price = price;
	}

	public int getYear()
	{
		return year;
	}
	public void setYear(int value)
	{
		this.year= value;
	}

	public int getMonth()
	{
		return month;
	}
	public void setMonth(int value)
	{
		this.month= value;
	}

	public String getTitle()
	{
		return title;
	}
	public void setTitle(String value)
	{
		this.title = value;
	}

	public double getPrice()
	{
		return price;
	}
	public void setPrice(double value)
	{
		this.price = value;
	}
}

The flexibility of the restful API calls  is pretty amazing. The first example (lines 62-67) shows how easy it can be to call a restful service.  Nothing is all that surprising as the server is passing back text and that text is assigned to a string.

The second example is slightly more useful as the return data is XML formatted and it is possible to convert that into a java object using JAXB.  XML data is still text data and it shouldn’t be amazing that text data is returned and assigned to a string.  Manually parsing out the data from an XML structure is really painful so I created the method convertXmlToMag to parse the XML into a java object.

The third example is by far the most useful example.  The restful server returns XML data but when making the restful call instead of passing in a string class I pass in a magazine class.  Thus the restful call returns the magazine object directly.  This also isn’t magic as the class does need to be defined when compiling the client but that is pretty much to be expected.  If the server does have certain types of data you wish to query then the client would already have similar or identical structured objects.

raw (text)data
Hello World
Having a great time!

200
<Magazine><Year>2016</Year><Month>6</Month><Title>Economist</Title><Price>9.99</Price></Magazine>

xml text data
year 2016
month 6
price 9.99
title Economist

pure xml object
year 2016
month 6
price 9.99
title Economist
Download files
Posted in programming, Setup From Scratch | Tagged , , | Comments Off on Restful services in Java – simple client example

Restful services in Java – simple server example

There are probably a lot of ways to create web services programs but the simplest way I could think of was to create a project using Eclipse and Jersey.

It turns out that restful services are really easy to summarize but a bit more involved to describe in the way of a teaching tutorial.

I have read up on the topic in the O’Reilly published book “RESTful Java with JAX-RS 2.0” by Bill Burke.    There is a lot of information starting with how to do this the hard way from scratch all the while moving on to how JAX-RS and markup can simplify the task.

I will not summarize everything that is described in the book but I will say that it if you want to get a working knowledge of restful services this is a pretty easy way to go about it.  The book is pretty focused on the server side itself.  The nice thing about creating a server like this is that it is easy enough to test using a simple web browser.

I have gone a small bit further and my examples will include some client code that will actually use the services that I created.  This is was done both as a simple java client and also with a simple html client that also used a bit of java script.

It was this last part that actually seemed the most interesting as you can then control the entire “application” from server to client.  The book uses xml examples but I did also get do some research on both java script, JSON and html/css.  It is possible that these are also highlighted in the future but at the moment they are not of a great interest to me.

 

Restful Services Setup from Scratch

There are a number of things that need to be setup for creating a web service and there are a number of ways to get this properly setup.  The way I choose was to create a new “Dynamic web project” in eclipse.

I created a new workspace to ensure that all setup needs to be done from scratch.

In this first dialogbox you can see that the “Target runtime” is an empty list.  This is because no web server has been defined.

Simply click on new runtime to define one.

Simply select which version of Tomcat is installed on your computer. The version that I had installed was 8.5.11.

Basically select the install directory of Tomcat.

Note: The structure of the Tomcat installation should be the same regardless of what the install directory is called.  However, when I was first trying to get the permissions for Tomcat setup I had problems using any path other than /opt/tomcat.  Once the permissions were corrected any path could be used.

Once the Tomcat server is defined, it is just a matter clicking through the next three dialog boxes.

We can set the context root of our URI to anything at here.This example it is FirstRestful.

Once you press Finish the rest of the structure is created.

Step 2 – adding Jersey

What is Jersey?  It is an implementation of JAX-WS (Java API for XML Web Services) for creating web services in XML format.  This makes it really easy for creating a small server.  All that needs to be done is to move the Jersey libraries into the correct spot under the WebContent folder.  Once this has been done then the basic framework is in place.

There is both a version 1 and version 2 of Jersey available on this site, newer must be better so of course I downloaded 2.25.1.

jaxrs-ri-2.25.1.zip

This zip file contained three directories full of jar files, add all of these to the project. Simply unpack the zip and put all of the jars files into the root of the WebContent/WEB-INF/lib directory.

Now that Jersey is added to the project all that is left is the source code.

Step 3 – adding the code

The last bit is to add the restful code to the project.  This first example actually does little more than display “Hello World”.

Ok, ok, I did get a bit ahead of myself and you can also see that I have created sayHelloXml and sayHelloHtml as well.  These were actually only used from the browser, you can forget about them for right now.

package com.acmesoft;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;

@Path("/Communicate")
public class HelloWorldEx {

    @GET
    @Path("/hello")
    @Produces(MediaType.TEXT_PLAIN)
    public String sayHelloTxt() {
    	return "Hello World\nHaving a great time!";
    }

    @GET
    @Path("/hello")
    @Produces( {MediaType.APPLICATION_XML})
    public String sayHelloXml() {
      	return "<Magazine>" +
		"<Year>2016</Year>" +
		"<Month>6</Month>" +
		"<Title>Economist</Title>" +
		"<Price>9.99</Price>" +
	"</Magazine>";
    }

    @GET
    @Path("/html")
    @Produces(MediaType.TEXT_PLAIN)
    public String sayHelloHtml() {
    	return "<h1>This is 'h1' heading</h1>";
    }
}

There is one other small file that needs to be setup, it is the web.xml file. This is the configuration file to setup the Jersey for this servlet.

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd" id="WebApp_ID" version="3.0">
   <display-name>User Management</display-name>
   <servlet>
      <servlet-name>Jersey RESTful Application</servlet-name>
      <servlet-class>org.glassfish.jersey.servlet.ServletContainer</servlet-class>
         <init-param>
            <param-name>jersey.config.server.provider.packages</param-name>
            <param-value>com.acmesoft</param-value>
         </init-param>
      </servlet>
   <servlet-mapping>
   <servlet-name>Jersey RESTful Application</servlet-name>
      <url-pattern>/*</url-pattern>
   </servlet-mapping>  
</web-app>

This example creates a small hello world application using restful web services.  You can see that the URI to get the hello world is /Communicate/hello, but this is only a part.

Eclipse does a lot for you.  One of the things it does is to create the context root of the URI. We can set it to any value we wish in the last step while creating our dynamic web project.

http://192.168.178.70:8080/FirstRestful/Communicate/hello

Step 4 – creating war file

The process of creating a war file couldn’t actually be much easier.  Simply right click on the project and select export and select war file.  This will open up the following dialog.

The destination in this case is /tmp.  The file is then copied to the /opt/tomcat/webapps directory in a second step. I could have set the permissions on the webapps directory to allow my user to write to it directly, however, that is not very security minded.  This isn’t an issue on a development machine but perhaps it is a bad habit to get into in general, thus the manual extra copy step.

The three checkboxes are reasonably clear to their purpose.  It is possible to optimize for a specific version of tomcat.  Export source files simply includes the source code in the war file.  This might be interesting for an internal application for your company but probably a bad idea if you are selling your application.

Step 5 – releasing application to tomcat

Releasing the application is as simple as copying the war file to the/opt/tomcat/webapps.

  1. stop Tomcat
  2. copy application to webapps directory
  3. start Tomcat

Tomcat will unpack the war file into the webapps directory.

 

Step 6 – viewing the result

Other configuration

In my opinion, this is actually a pretty good URI and isn’t nested too deep.  However, it is possible to add another level to it.  This additional level is defined in the web.xml file.

   <servlet-name>Jersey RESTful Application</servlet-name>
      <url-pattern>/newlevel/*</url-pattern>
   </servlet-mapping>

With this change, there would be another level added to my URI but that seemed to be excessive to me.

What might really be a better choice might be to change the root context.  If you do change your mind you can easily change this to anther value.This is done by simply changing the Web Project settings.  This can be selected from the properties menu (right click on the project and select properties).

This is all the steps you would need to do in order to create a working server.  This can easily be tested out using your web browser, or you can use other tools.  I will discuss that in the next step when I create the first client.

Download files

 

Posted in programming, Setup From Scratch | Tagged , , | Comments Off on Restful services in Java – simple server example

Restful services in Java

Over the years there have been different methods of for communicating between heterogeneous systems.  Each of these methods have to deal binary messaging.  Some of the different protocols that have been used are as follows.

  • Remote procedure calls (RPC)
  • Distributed Component Object Model (DCOM)
  • Common Object Request Broker Architecture (CORBA)
  • Simple object access protocol (SOAP)
  • Windows Communication Foundation (WCF)

These protocols are of varying complexity and some of the descriptions that have been used to describe them are complexity, rigid, complex, heavyweight and complicated.

Each time the technology wheel turns a new protocol is created with the goal to create something that is powerful and lightweight.

Restful web services is an attempt at providing a lightweight but scaleable communications protocol that can be called from any platform.  It uses a simple URL instead of creating and passing complicated XML structures

This should be a very simple way of submitting information or retrieving data.  A colleague once recommended that I create my own web services for processing my weekly expenses while I was on the road.

At that time, it seemed like overkill to create such a solution in order to process anywhere from 5 to 10 expense items per week.

  • Setup a computer with web server
  • Program a web service to accept the price data
  • Create some GUI process for entering the data

I did have the interest in the project but ended up focusing my efforts on dabbling in the hardware arena by creating a LED cube.  Eventually, my interest has come back to web services.

Of course as what often happens is that you cannot start something without doing something else first – in this case all the prerequisite software.

My laptop is a bit of a mess so I choose to create a small virtual machine and do all of my work there.  This is actually a extremely interesting use of those extra cpu cycles and could become common for even basic computer users in the future.  I hope to go into that at a later point.

For working with restful web services I needed to install the following.

Once everything was setup, I did get a chance to create both a a server and a client.  I will not actually setup my own restful server setup  for remotely doing my expenses but I may end up using the technology for something else in the future.

The next few blog entries create a few simple examples to put the technology through its paces.

Difference between Soap and Rest protocols.

http://blog.smartbear.com/apis/understanding-soap-and-rest-basics/

Posted in programming, Setup From Scratch | Tagged , , | Comments Off on Restful services in Java

a plethora of shells – what is my shell interpreter

I kept seeing odd error messages from my shell script that simply made no sense.  No matter what changes I made a new error seemed to pop up.  A short while later I did notice at the top of the script that it was using a korn shell interpreter.

#!/usr/bin/ksh

This was not as helpful as you might initially guess.  Sure there are differences between the two shells but most of the scripts that I write would work under either.  Besides, I had a small script with the new code I wanted to test and it worked fine under both the bash and korn shell.

I didn’t initially consider that the the script itself wasn’t not being run in the same manner by the calling system as by me.

In the end I wanted to determine if the shell was the same as what I had selected – after all how could it not be my shell.

The most straight forward method would be to see which shell is interpreting.  It is just as easy as checking which process or processes are running.  Just to see which programs, in this case our shell, is running I ran the process status command (ps).

Nothing else is really running in my Korn shell example.  The only two processes is the korn shell and the ps command.  The Korn shell is not really all that surprising as this is a terminal window for my user and the Korn shell was the shell setup for in the /etc/passwd file.

Korn shell

> ps -f

     UID   PID  PPID   C    STIME TTY         TIME CMD
   fkadm  3819  3816   0 14:52:41 pts/2       0:00 -ksh
   fkadm  4609  3819   0 14:54:37 pts/2       0:00 ps -f

It is almost the exact same story for the Bash shell.  My user was started with a Korn shell and (process 23509) and then ran the Bash shell (process 23541).  The ps command is then run from the Bash shell reporting on anything that is running.

Bash shell

> ps -f

     UID   PID  PPID   C    STIME TTY         TIME CMD
   fkbin 23509 23507   0 10:27:31 pts/4       0:00 -ksh
   fkbin 23541 23509   0 10:27:31 pts/4       0:00 bash
   fkbin  4128 23541   0 14:53:51 pts/4       0:00 ps -f

The starting point for my finding my shell is to know which of the many processes to look at.  The good news is that I am able to get my process id using the special shell variable “$$”.

Everything from that point is pretty straight forward.  We can use the -p and -o arguments with the ps command to see specific values for a specific process.

 

process status (ps)

argument Description Description
-p process id selects process status from list where the process id matches this value.
-o format The format is one of many possible fields that can be selected from the process status.  There are quite a few possibilities, but the ones that stood out as the most immediately interesting are as follows.

  • user
  • pid
  • ppid
  • stime
  • tty
  • args
  • comm

The first six should look familiar as these are the values that are output when doing a full listing (-f).  The comm format is the actual command which in this case is the interpreter while the args is both the command and its arguments.

With this little bit of knowledge I had enough information to query what interpreter was running my script.

#!/usr/bin/bash
MYPID=$$
MYSHELL=`ps -p $MYPID -o comm | egrep -v "CMD|COMMAND"`
MYCOMMAND=`ps -p $MYPID -o args | egrep -v "CMD|COMMAND"`
echo the shell for process id $MYPID is $MYSHELL.  
echo the entire command is $MYCOMMAND

This type of shell goodness should be all that it takes to find out more about how you are being called.  I was a bit surprised to see that my script was being called with a different shell altogether.

The shell calling my program was /usr/bin/sh which is a lot less compatible with either the Korn or Bash shell.  A bit more digging and I found out that the script was being passed directly to the shell interpreter.  This explains why the Korn shell wasn’t being used.

sh /home/specialuser/scripts/myscript.sh

Well, I guess this is proof that you cannot always trust IT.

Posted in Command line | Tagged , | Comments Off on a plethora of shells – what is my shell interpreter

Windows powershell scripting

Powershell scripting is not too difficult to pick up and there is a wealth of examples on the internet to help solve any syntax related question.

Simply call the appropriate Powershell “cmdlet” to perform the required task.  The return value can be manipulated directly without being forced to assign intermediate values to a temporary variable.  It is easy enough to manipulate or format the return value.

$datestr = (get-date).tostring('yyyyMMdd')

This example uses the get-date cmdlet to get the current date and format it into reasonably friendly way.

Functions

Just like any proper language it is possible to create functions to assist in breaking up program logic.

Powershell has a number of primitives that can be used for creating functions or other program logic.  This list of supported primitive is actually fairly large but here is a smaller subset which covers the usual suspects.INT

  • BOOL
  • FLOAT
  • DOUBLE
  • STRING

Microsoft has taken a route less traveled with respect to how they define a function.  The parameter types are surrounded with square brackets. In addition, they seem to have gone back to “BASIC” and put dollar signs before each variable.

# write out log info
Function My-Add-Function([float]$firstval,[float]$secondval) 
{
	$returnval = $firstval + $secondval
	return $returnval
}

Microsoft allows you to call your user defined functions in the way you would expect.  You simply call the function with the values being passed.

ie.

	my-add-function $firstvalue $secondvalue

Yet it is also possible to actually pass in the parameters in any order if you simply give the name of each parameter as they are named in the function.

ie.

	my-add-function -secondval $secondvalue -firstval $firstvalue

I don’t know of any other languages or scripts that allow you to call your own functions in this manner.  It is interesting but I don’t see it as an important feature.

It actually took me a while to see what was missing from the function declaration – there is no return type.  This doesn’t mean you cannot return values from functions but how it is done is odd to say the least.

# function GetDateNow returns todays date
Function GetDateNow 
{
        $TimeStringNow=(get-date).tostring('yyyyMMdd')
        return $TimeStringNow
}

Basically all output that occurs in the function is returned.  Looking at this method you would think that the variable TimeStringNow is the only thing being returned and it is because of the return keyword.  This is not the case.

# function GetDateNow returns todays date
Function GetDateNow 
{
        $TimeStringNow=(get-date).tostring('yyyyMMdd')
        $TimeStringNow
}

This second function declaration is semantically identical to the first.  The return keyword simply exists so it is possible to exit a function at a specific point, it has no connection with the values being returned.

The one part about the return values that I didn’t mention is that everything that is output in the function (no matter when) will be returned to the caller.

I actually do not like how data is returned from a function but I cannot say that this is completely unique.  This is similar to how Unix shell scripting works if you call a function while capturing the output from that function to a return variable.

#!/usr/bin/bash
function unixfunction 
{
echo "here at point 1"
echo "here at point 2"
echo "here at point 3"
echo returning 6
echo 6
}

retval=`unixfunction`
echo $retval

The output looks like this.

here at point 1 here at point 2 here at point 3 returning 6 6

I don’t really like Powershell scripting but that might just be an anti-Microsoft bias. Using Powershell is actually miles better than using batch scripts from all those years ago. It is really just a matter of taste, if you are a Windows aficionado this is not only a good choice but perhaps the best choice as it comes standard with Windows.

Does it suck?  Well, I guess not.  If you look at this from the perspective of replacing the cmd.exe with Powershell you are probably trading up.  If you want more command line programs that you string together then in my opinion it isn’t really an improvement.

 

Posted in programming | Tagged , | Comments Off on Windows powershell scripting

Windows scripting sucks or does it?

I am not a fan of the big MS.  It started a long time ago when my funds were limited and I simply couldn’t afford to keep up with the MS upgrade schedule.  Not all of their software back in the day, in my opinion, was all that great.  I went cold turkey on as many of the MS products that I could.

There is a lot of open source software that could fill in the empty spots for me and since then most of my work life was on Unix which led to joys of shell scripting (ksh,bash,csh)

I love Unix shell scripting because the shell is basically just the glue to tie together a lot of small command line utilities which makes it easy to create and add extra commands, and Unix has a lot of command line tools.  Windows, at that time, had a very weak batch scripting language and a limited number interesting command line commands.

PowerShell

(Eventually) Those folks at MS decided to create their own scripting language called powershell.  They decided to take a slightly different route than the Unix crowd. Powershell is an interpreter that contains the logic for doing tasks.  This “shell” appears to be more of a replacement for the typical cmd.exe shell that has existed since the beginning (well it used to be command.com) as well as all of the command line programs.

The commands that exist from the powershell are called cmdlets.  Unlike Unix these are not a simple executables that can be executed from the cmd shell but are commands that are part of the powershell interpreter.  This might give a small bit of extra performance but it also means a new version of powershell is needed to add some new functionality.

The one good thing about the powershell is that it was installed on my computer by default. In addition, The big MS did want to show their security credentials and they have added a security level that must be set before you are able to run any powershell scripts.  This security level is essentially determining to which extent you trust powershell scripts with the focus being on either local or from other sources.

This is actually a pretty smart as it is possible to do quite a few things using powershell once you can run the scripts.  It would be pretty serious to have such power unlocked by default for any user who happens to execute a powershell script that is an attachment to an email.

However, even accidentally running a powershell script is a bit harder as MS has associated the extension .ps1 to be assigned to notepad.  Thus if you click on a script from windows explorer you will simply open up the script in notpad.

In order to run powershell scripts you need to first select an executable permission. There are four different permissions that can be selected in order to set the various rights at different levels.

Level Description
Restricted Can not run powershell scripts
RemoteSigned Can run local scripts but remote scripts need to be signed.
Unrestricted Any script can be run.
AllSigned Very secure, requires all scripts to be signed.
Undefined Removes existing level

Changing the level is actually pretty easy but you do need to be an administrator to do so. Simply use the set-executionpolicy cmdlet to set the permission to one of the described levels.

Signed Scripts

I personally am a fan of the unix “permissions” as it is really quite simple and very easy to modify. However, I wouldn’t say that they are the final word in preventing unauthorized scripts from being run.

I guess the final word is trying to protect your scripts is to use Microsoft’s method of creating recognized certificates while using a certification authority infrastructure.  If you want you can learn more about this by visiting Microsoft and downloading their best practices document.  I didn’t feel up to the task of going through the entire 67 page document so instead I decided to set my execution policy level to RemoteSigned.

This then let me run my fabulous powershell script.

$list = dir
echo $list

Actually this small script does quite a bit.  It does a directory listing and assigns the output to a variable.  Much like the bash shell that variable can be a list and echoed to the output device.

 

Note: It is possible to run any other programs from this powershell if they are in your path.

Unix friendly

It is possible that the big MS wanted to create a shell that would also appeal to the Unix crowd. I think this is the case as the powershell has the alias command.  This gave them the ability to create a bunch of commands that appear to include the common Unix commands.

The alias command links the name to an internal cmdlet which performs some action. One example of this is the “clear” alias being linked to the cmdlet clear-content.  The alias’s that they choose appear to look like they were picked to make Unix guys feel comfortable.

PS C:\Windows\system32> alias

CommandType     Name                            Definition
-----------     ----                            ----------
Alias           %                               ForEach-Object
Alias           ?                               Where-Object
Alias           ac                              Add-Content
Alias           asnp                            Add-PSSnapIn
Alias           cat                             Get-Content
Alias           cd                              Set-Location
Alias           chdir                           Set-Location
Alias           clc                             Clear-Content
Alias           clear                           Clear-Host
Alias           clhy                            Clear-History
Alias           cli                             Clear-Item
Alias           clp                             Clear-ItemProperty
Alias           cls                             Clear-Host
Alias           clv                             Clear-Variable
Alias           compare                         Compare-Object
Alias           copy                            Copy-Item
Alias           cp                              Copy-Item
Alias           cpi                             Copy-Item
Alias           cpp                             Copy-ItemProperty
Alias           cvpa                            Convert-Path
Alias           dbp                             Disable-PSBreakpoint
Alias           del                             Remove-Item
Alias           diff                            Compare-Object
Alias           dir                             Get-ChildItem
Alias           ebp                             Enable-PSBreakpoint
Alias           echo                            Write-Output
Alias           epal                            Export-Alias
Alias           epcsv                           Export-Csv
Alias           epsn                            Export-PSSession
Alias           erase                           Remove-Item
Alias           etsn                            Enter-PSSession
Alias           exsn                            Exit-PSSession
Alias           fc                              Format-Custom
Alias           fl                              Format-List
Alias           foreach                         ForEach-Object
Alias           ft                              Format-Table
Alias           fw                              Format-Wide
Alias           gal                             Get-Alias
Alias           gbp                             Get-PSBreakpoint
Alias           gc                              Get-Content
Alias           gci                             Get-ChildItem
Alias           gcm                             Get-Command
Alias           gcs                             Get-PSCallStack
Alias           gdr                             Get-PSDrive
Alias           ghy                             Get-History
Alias           gi                              Get-Item
Alias           gjb                             Get-Job
Alias           gl                              Get-Location
Alias           gm                              Get-Member
Alias           gmo                             Get-Module
Alias           gp                              Get-ItemProperty
Alias           gps                             Get-Process
Alias           group                           Group-Object
Alias           gsn                             Get-PSSession
Alias           gsnp                            Get-PSSnapIn
Alias           gsv                             Get-Service
Alias           gu                              Get-Unique
Alias           gv                              Get-Variable
Alias           gwmi                            Get-WmiObject
Alias           h                               Get-History
Alias           history                         Get-History
Alias           icm                             Invoke-Command
Alias           iex                             Invoke-Expression
Alias           ihy                             Invoke-History
Alias           ii                              Invoke-Item
Alias           ipal                            Import-Alias
Alias           ipcsv                           Import-Csv
Alias           ipmo                            Import-Module
Alias           ipsn                            Import-PSSession
Alias           ise                             powershell_ise.exe
Alias           iwmi                            Invoke-WMIMethod
Alias           kill                            Stop-Process
Alias           lp                              Out-Printer
Alias           ls                              Get-ChildItem
Alias           man                             help
Alias           md                              mkdir
Alias           measure                         Measure-Object
Alias           mi                              Move-Item
Alias           mount                           New-PSDrive
Alias           move                            Move-Item
Alias           mp                              Move-ItemProperty
Alias           mv                              Move-Item
Alias           nal                             New-Alias
Alias           ndr                             New-PSDrive
Alias           ni                              New-Item
Alias           nmo                             New-Module
Alias           nsn                             New-PSSession
Alias           nv                              New-Variable
Alias           ogv                             Out-GridView
Alias           oh                              Out-Host
Alias           popd                            Pop-Location
Alias           ps                              Get-Process
Alias           pushd                           Push-Location
Alias           pwd                             Get-Location
Alias           r                               Invoke-History
Alias           rbp                             Remove-PSBreakpoint
Alias           rcjb                            Receive-Job
Alias           rd                              Remove-Item
Alias           rdr                             Remove-PSDrive
Alias           ren                             Rename-Item
Alias           ri                              Remove-Item
Alias           rjb                             Remove-Job
Alias           rm                              Remove-Item
Alias           rmdir                           Remove-Item
Alias           rmo                             Remove-Module
Alias           rni                             Rename-Item
Alias           rnp                             Rename-ItemProperty
Alias           rp                              Remove-ItemProperty
Alias           rsn                             Remove-PSSession
Alias           rsnp                            Remove-PSSnapin
Alias           rv                              Remove-Variable
Alias           rvpa                            Resolve-Path
Alias           rwmi                            Remove-WMIObject
Alias           sajb                            Start-Job
Alias           sal                             Set-Alias
Alias           saps                            Start-Process
Alias           sasv                            Start-Service
Alias           sbp                             Set-PSBreakpoint
Alias           sc                              Set-Content
Alias           select                          Select-Object
Alias           set                             Set-Variable
Alias           si                              Set-Item
Alias           sl                              Set-Location
Alias           sleep                           Start-Sleep
Alias           sort                            Sort-Object
Alias           sp                              Set-ItemProperty
Alias           spjb                            Stop-Job
Alias           spps                            Stop-Process
Alias           spsv                            Stop-Service
Alias           start                           Start-Process
Alias           sv                              Set-Variable
Alias           swmi                            Set-WMIInstance
Alias           tee                             Tee-Object
Alias           type                            Get-Content
Alias           where                           Where-Object
Alias           wjb                             Wait-Job
Alias           write                           Write-Output

PS C:\Windows\system32>

This is NOT exactly like Unix. The “ls” alias does do a directory listing but unsurprisingly it does not accept the same options.

Microsofts cmdlets

PowerShell actually implements all the logic in the form of cmdlets.  You could find a book that described PowerShell programming along with (hopefully) a big reference section. That is one possible approach but the good news is that PowerShell has built in help.

The cmdlet get-help will return the information about any of the supported cmdlets.

Microsoft also has a very complete webpage that describes the different cmdlets. This information is grouped together by function.

https://technet.microsoft.com/en-us/library/dd772285.aspx

One final note about security

Well, I am glad that I didn’t spend a whole lot of time on setting up an elaborate system to deal with signed certificates.  It turns out that there are actually a number of different ways that you can run powershell scripts and some of them get around the existing security settings.

I didn’t actually come up with these methods myself, but instead happened upon the following webpage that elaborated a lot of ways to get around the execution policy.

15 Ways to Bypass the PowerShell Execution Policy

Of all the methods of bypassing the security I liked the method of piping the script through the interpreter and telling it to ignore the execution permissions by not using anyone’s profile.

Does windows scripting suck?  It is a bit too early to answer this question as the only script I have provided is equivalent to the dir command.  I will cover a bit more about powershell scripting in part II.

Posted in programming | Tagged , | Comments Off on Windows scripting sucks or does it?

high quality USB cables

He said          I don’t want a tablet

She said         They are really convenient.

I should probably check my wallet to see if I belong to some sort of anti-technology society. I actually didn’t have a tablet for the longest time and then eventually after a lot of thought I got a tablet.

Well, that’s a bit of a lie.  I agreed with my wife that it would be an ok for her to get me one as a Christmas gift – although what would better would be a nice pair of socks.  I didn’t get the socks and the tablet didn’t really displace any computers.  Yet, I did find the tablet to be useful for looking up things on the internet and doing some reading.

It must have wormed its way into my heart as I became a bit angry that it might need to be replaced after a few years.  The real problem was it wasn’t charging or couldn’t hold a charge.  It might take three days to get fully charged for just a few hours.  Did it get kicked?  One fall from the bed too many?

I should have suspected either the charger or the cable right away.  After all, sometimes when my call phone was charging I could hear it connect and disconnect quite a few times over the course of the evening.

I was tidying up my shelf while the table was in the middle of one of its marathon charging sessions.  I must have bumped it as I then heard it disconnect but before I could do anything it reconnected.

Well, now that interesting…

A small bit of upwards pressure on cable? (kaplink) Tablet charging.

A small bit of downwards pressure on cable? (kerplink) Tablet no longer charging.

repeat a few times

This didn’t rule out a problem with the tablet but it convinced me that the charger was ok.  A quick switch of cables ruled out a problem with the tablet itself.

I must live a pretty sheltered life as I haven’t had a problem with a USB cable not doing my bidding.

I try and buy good cables and good chargers and apparently so far I have been lucky. However, once I make the upgrade to USB 3.0 I will need more than to rely on good luck as those cables carry enough power to fry just about anything plugged into it.

Amazon bans faulty usb c cables – Google engineer reviewed hundreds

https://www.theguardian.com/technology/2016/mar/30/amazon-clamps-down-dangerous-usb-c-cables

Apparently, the cheapest isn’t always the best.  I guess you get what you pay for and in the future, saving a few pennies on a cable might get you the opportunity to purchase a new tablet or phone.

Posted in Soapbox | Comments Off on high quality USB cables