Security in homogeneous systems

The Central Intelligence Agency (CIA) has been quite industrious in creating a new way[1] of tracking documents that could be “borrowed” from a person, business or government against the will of of that entity.

This new document protection is the creation of a watermark that is also a web beacon in the document.  If you think about it, this is a pretty neat way of “bugging” a document to call home when it is opened.

This particular trick works (at the moment) only on Microsoft Office files.  Not a very surprising choice considering the market penetration of MS office.  I cannot find any usage statistics for MS Office but it seems reasonable that Microsoft has at least 50% of the overall market including personal usage and business usage.  I would bet that the business usage for medium and large sized company’s is much much higher.  There are some big firms and governments that use open source office suites but many more that use Microsoft Office solutions.

However, those ambitious CIA fellows must have had a blind spot in considering that people would only use MS products.  This probably doesn’t work when you open these documents with other office suites.  Not only that this foreign intrusion may actually be visible.  Oops.  I guess once they think this through to the end they will realize that was a bit of a silly assumption on their part.

Heterogeneous for the win

It is really a lot easier to support a company or division when everything is the same.  You have a few default images for the few types of systems.  In this situation, setting up a computer is really easy.  Really easy is not always the best solution.

All software has bugs and it is only a matter of time before some hacker or state agency finds a zero day bug they can exploit.  If all systems are using the same operating system then once a bug is found it would allow the hacker the ability to exploit it everywhere.  If some of the systems utilize a different operating system they would form type of firebreak against the exploit and help to make it safer.

Having different systems or programs can make support a bit harder but it will add subtle layers security at the minimum.

I thought that there were more articles about the dangers of homogeneous systems but Google cannot seem to find them.

Well, other than this white paper.

Read more: Whitepaper: Attack of the Clones – Is Homogeneity in a network environment safe?

 

Posted in security | Tagged , , | Comments Off on Security in homogeneous systems

The power of mathematics in government

DEBT

This is a word that you rarely hear mentioned much by politicians especially if they are campaigning for office.  Oh, sure you might hear something about how they will fix the deficit.  The deficit that the other guys have allowed to run rampant will be fixed.

Oh, sure we will cut the deficit by about 10% a year and then in about 10 years we will have that nasty deficit tiger tamed.  Usually the savings are conjured up by playing up how growth and thus tax receipts will be increased.

During an election when people or groups claim that these savings are either unlikely or not realistic.  These nay sayers are usually accused of being partisan.  Indeed this might be true but considering how optimistic politicians are (every single time) and how often they are incorrect – again virtually every single time maybe some pessimism is warranted.

How big of a deal is a 500 billion dollar deficit when compared to a 19.8 trillion dollar economy?  Well, people have been hearing about the deficit so much they have forgotten that it really means unpaid bills or borrowed money.  These small (ha) deficits are what keep adding up to mountains of debt.

budget deficit when your expenses exceed your income.
debt the sum of all of the money owed.  these values are essentially the sum of all budget deficits.

It would take a real concerted effort to pay any large debt it off and downright impossible without reducing the budget deficit to zero.

But it this really a problem? Perhaps it isn’t a problem with these low interest rates.  The Fed rate is currently at 1/2%.  In one sense this debt bomb is less of a problem with low interest rates but it isn’t the principle that will be the eventual problem but the interest.

I considered writing up a spreadsheet to try and describe the future evils that will be visited upon the US, but the Internet Wallstreet Journal has already done some looking into this issue.

https://blogs.wsj.com/economics/2015/02/03/the-legacy-of-debt-interest-costs-poised-to-surpass-defense-and-nondefense-discretionary-spending/

If you choose not to read the article, then at least read the headline.

The Legacy of Debt: Interest Costs Poised to Surpass Defense and Nondefense Discretionary Spending

This type of problem is not only limited to the US, one of the key weaknesses of the Maastricht was that deficits needed to be limited to 3%.  Limited to 3% pretty much gives the EU members the option of never really balancing their budget.

Politicians have a tendency ignore anything that they really don’t wish to address.  The key sentence for the Europeans.

annual deficits no greater than 3% of GDP

Sure, 3% is a small number but if it is left to grow at this amount even a 3% growth will cause the debt to double in about 25 years – assuming no unexpected circumstances increase the deficit.  Well, I am sure that the EU countries are making sure that in some years they have a surplus – right?

I am a tech guy (not a banker) but it is pretty disappointing to occasionally hear comments from people who cannot tell the difference between deficit and debt, and (US) politicians that play on that.

There will be some real problems for big countries that have never had to make hard decisions on their budgets if things don’t change.

Tell your friends the difference, tell your family the difference or perhaps most important tell your politician the difference (and that it matters to your children and their children).

 

Posted in Soapbox | Comments Off on The power of mathematics in government

… and because perception matters

“I am from the government and i am here to help”

                                            Ronald Reagan

It has certainly been a while since the government or the career politician has received a lot of respect.  I don’t find any statistics going back more than a few decades but it turns out that US politicians, especially congressman, end up much lower in the ranking over time.

Congress approval ratings in 2017             20%

Congress less popular than                           France

I don’t really want to defend the career politician but in this day and age their job is much tougher than when some of their predecessors such as when Abe Lincoln was in office. Actually, today it is even much harder than some of the more recent office holders such as Reagan or Clinton.  With the rise of technology their previous votes, speeches, or other misdeeds that have occurred in the last months, years or decades now come back to haunt them – sometimes in the form of video.

If you thought it was tough living down something you did in your youth imagine your every move being recorded for later use by groups just looking to find discrepancies or “flip flops” of opinion.  In addition, people or groups with personal agenda’s use all of this information with the intent of causing a politician to look stupid or clueless.

If you think that dealing with facts if difficult then try to imagine dealing perception.  It is just as powerful but not always grounded in facts.  There is even a trading adage to this effect.

Buy the rumor, Sell the fact

It is because perception is so powerful that small companies as well as large companies struggle hard to shape perception.  No CEO wants to loose customers or market capitalization because the public perceives them to be ambivalent about some topic or other – perception matters.

James Comey may have been fired for a lot of reasons but the reason that most people are looking closely at is that he was investigating the Trump campaign into collusion with Russia.

Were there any good reasons for firing James Comey?  Probably. What this the right time to do so? I suspect that historians will be answering these questions for years to come but this firing could be adding to the public perception that President Trump has something to hide.

This one firing would not necessarily matter so much if it was the only thing that has occurred during President Trump’s brief tenure as President.

In fact President Trump should be happy that special prosecutor Robert Mueller has been brought in to finish this task.  If for no other reason than it helps to refute the perception that there is anything that needs to be hidden, because perception matters.

It actually does not help President Trump’s case when he is exercising his hyperbole in some of his statements about this current situation.

The people who were happy with President Trump may or may not be less happy with him now. For those people who either did not like him before or had neutral feelings the various storms whirling around the White house do make a difference in how he is perceived as well as how America is perceive abroad.

I could take a cheap shot and find some reference from Donald Trump the campaigner to try and make him look less dignified or poke fun at him.  Instead, perhaps he will take some advice from someone he truly does trust.

Well, because perception does matter.

 

Posted in Soapbox | Tagged | Comments Off on … and because perception matters

professionalism in IT … at the movies

My first boss was a real ball-breaker.  He had a very good idea of what was possible with computers, what the users might need and how to achieve that.   He could have literally done it all himself except there were not enough hours in the day – so he needed staff.  When his staff had problems and needed assistance you could come to him for help.

First and foremost, when you show up, wait for him to finish what he was working on.  At the time it seemed like a power move but after years of reflection it was so he could finish whatever debugging he was in the middle of.  It takes longer than you would think to be in the zone and it is easy to get knocked out.

He would have expected that you had at least done your homework.

  • be able to clearly describe the problem
  • have the exact error message at hand
  • demonstrate what you have done in an attempt to correct this problem
  • be prepared to describe what you have done
  • know the code you bring to him

If you weren’t prepared the meeting probably wouldn’t go all that well.  He helped to cement how I would work on problems and what level I would expect from my colleagues.

I have a tendency to think about him whenever I see what I consider to be stupidity in the workplace.  I was surprised when I almost heard his voice at the movies.

It seems sloppy that in 2016 computers and point of sale terminals should be providing less information now than they probably did in the 1950’s.

cinestar-receipts-concessions

Concession receipt for 2 sodas and some popcorn

It my be difficult to see but this scanned receipt is a reasonably high res receipt that includes special symbols, German special characters, and bar codes.

I am a bit surprised how the number of items is represented.  I personally don’t think that the concession stand will be selling double digit of anything much less four digits of anything.  Someone decided they want all of their numbers to align so that isn’t the oddity but the leading zeros are.  Perhaps that is just a personal dislike.

The receipt itself actually has more information about the company and the branch than the food.  I know that snacks at the movies are expensive but from looking at this receipt it is almost as if the movie theater is embarrassed to say just how expensive each item truly was.

It is a possibility that receipts in Germany are more of the simple variety, but alas that isn’t the case.

tedi-receipt

receipt for USB cable and a flashlight

 

This is a receipt from a store that would be most like a dollar store in the USA.  This receipt contains as much information about the store in general as well as a lot of information about what was purchased.  The item #, description and cost per item is all clearly displayed.  Closer to the bottom you can even see how much of your purchase went to the government in the form of taxes.  This is a great receipt.

Taking one final look at my concession receipt

Dies ist keine Rechnung im Sinne des §14 UStG…

Well, I guess the receipt that I asked for is a receipt but not a receipt in the sense of what is a receipt as described by a long and dry law description clarified by §14 of UStG.

It is not like this particular movie theater cannot produce more complicated output.  They did a much better job with the tickets – font size, German characters and QR codes.

cinestar-receipts-seats

These are actually both the receipt and the ticket.  Plenty of information here as well including a nifty QR code which probably contains all of this and more.

If worked on the concessions output then my boss would be pulling me aside right now for a small talk.

Posted in Soapbox | Comments Off on professionalism in IT … at the movies

Is Agile development the Emperors new clothes

I had the opportunity to take a class on Agile development recently, Scrum, and it was a fascinating way of doing development. I have been pretty lucky over my career that I have not been dragged through a lot of heavy development processes. Yet, while working on some projects I have experienced the joys of slow processes, excessive meetings, indecisive users, as well as plenty of scope changes. Management would blame us if we didn’t deliver on time something that was completely different than their original program.

Initially while sitting in my class I was thinking that Scrum might be the solution to all those problems.

No specific due date is given for delivering on those features, just a series of releases each with more functionality than the last. There is no real penalty for change, well not for the developer; Perhaps for the product manager depending on the company culture. You even have your own advocate to deal with either incidental or structural problems.

About the time that I was convinced Scrum was the solution to all problems everywhere that I started to have a few doubts. I personally am not a big fan of writing up specifications or user documentation and the people have a tendency to not read it much either. This much unloved type of documentation is however requested by auditors.

I asked about this in class and oddly enough documentation was not one of the artifacts of the process. Sure, you could define “done” to include this but developers are not natural writers and from a productivity standpoint you would really want them to write code. What about adding a layer around the scrum team of technical writers?

That might be just as bad as not doing scrum in the first place. You will have a constant flow of people waddling over to the developers breaking their concentration.

Then why does Scrum work?

  • The developers set their own due dates rather than management
  • A product manager is available to answer all questions about the tasks being developed
  • The product manager can make decisions regarding the direction of the product
  • The number of meetings for the most part are all defined and limited
  • The developers are expected to be capable, independent and have courage
  • The scrum master is dedicated to eliminating impediments to team progress
  • The list of features have been ordered by priority and the important ones should be well defined.

The transparency helps management to realize that with the resources in place some of their requests are not possible.

What development methodology wouldn’t be better with these as preconditions? If you have dedicated group of smart people doing the work who are able to get feedback on their work along with someone fighting to remove difficulties then you will see good results.

Just like all other processes in the world the better your inputs the better your final results.  These results could be faster, smaller or less bug laden.  I guess this could be summed up with use better staff and try and stay out of their way … well, that’s my thoughts on agile.

 

 

Posted in Soapbox | Tagged | Comments Off on Is Agile development the Emperors new clothes

Restful services in Java – 360 degree of restful client

Always try and use the most appropriate tool for the task.  The Apache HTTP server or web server is a program for serving the html web pages to the client who requests them. Tomcat is a application server that is actually used to for serving up Java Servlets or JSPs that are packaged up in a Web Archive file (war).

That isn’t to say that Apache cannot serve up Java Servlets nor does it mean that Tomcat cannot serve up static html pages.  Both programs exist because each is specialized for their particular task.

However, in order to create my web client I am going to have Tomcat serve up both the application as well as the static html web page that calls this service.  In order to do that, I will need to make a small change to my Tomcat configuration so it can also serve up these static pages.

Static Setup

Tomcat is configured by default for Java Servlets but by adding a tiny bit of extra configuration it will then serve up the static pages.

In the Tomcat configuration directory /opt/tomcat/conf is the server.xml file which contains a lot of the configuration for Tomcat.  Simply adding the following line to the Host block will define the location for the static web pages.

<Context docBase="/opt/tomcat/webapps/test" path="/test" />

The docBase attribute points to the path where the static html will be located while the path attribute is the prefix for the web page.  With the above setup, the following would be the url to access the index.html page.

http://localhost:8080/test/index.html

This block actually contains more interesting configuration that should probably be explained.

 <Host name="localhost" appBase="webapps"
             unpackWARs="true" autoDeploy="true">

    <!-- SingleSignOn valve, share authentication between web applications
         Documentation at: /docs/config/valve.html -->
    <!--
         <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
    -->

    <!-- Access log processes all example.
         Documentation at: /docs/config/valve.html
         Note: The pattern used is equivalent to using pattern="common" -->
    <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
           prefix="localhost_access_log" suffix=".txt"
           pattern="%h %l %u %t &quot;%r&quot; %s %b" />

    <Context docBase="/opt/tomcat/webapps/test" path="/test" />
 </Host>

The autoDeploy attribute gives the flexibility of deploying a war file while Tomcat is running.  This is actually pretty neat, you don’t need to stop and restart the server when installing the new war files.  Simply copy the war file to the webapp directory.

While doing some of my testing I simply would remove the war file and the directory that was unpacked from it.  After that I would copy over the new war file and wait for a few seconds.  At this point I could then call my restful service again.  Tomcat would unpack the new war file and then run the application.

Web Client

My client is really lightweight.  It simply queries the starting index and count of how many items should be retrieved.

Simply enter the values that you are interested in and click on the “fetch data” button.

The actual html for the webpage itself is trivial.  It simply makes the restful call and then parses the return list and displays each item.

The tricky part is not displaying the fields, gather the input or the button but actually the java script behind the webpage to make the AJAX call.  I am not a web developer so I will just try and point out the interesting things for the AJAX call.

360 Web Client Code

<!DOCTYPE html>
<html>
 <head>
 <meta charset="UTF-8">
 <title>JavaScript &amp; jQuery - Chapter 8: Ajax &amp; JSON - Loading HTML</title>
 <script src="js/jquery-1.12.1.js"></script> 

 <script>
 function displayname() {
 var start = document.getElementById("startidx").value;
 var count = document.getElementById("count").value;

  var xhr = new XMLHttpRequest(); // Create XMLHttpRequest object
 var myurl = "";
 myurl = 'http://localhost:8080/ThirdRestful/price/fetch?start=' + start + '&count=' + count;

  xhr.open('GET', myurl, false); // Prepare the request
  xhr.send(null); // Send the request
 
  responseObject = JSON.parse(xhr.responseText);
 
 var newContent = '';
 newContent += '<table align="left">';
 newContent += '<tr>';
 newContent += ' <th align="left">Id</th>';
 newContent += ' <th align="left">Symbol</th>';
 newContent += ' <th align="left">Bank</th>';
 newContent += ' <th align="left">Price</th>';
 newContent += ' <th align="left">Exchange</th>';
 newContent += ' <th align="left">Quote Timestamp</th>';
 newContent += '</tr>';

 for (i = 0; i < responseObject.records.length; i++)
 {
 newContent += '<tr>'; 
 newContent += '<td width="55px">'; newContent += responseObject.records[i].id; newContent += '</td>';
 newContent += '<td width="95px">'; newContent += responseObject.records[i].symbol; newContent += '</td>';
 newContent += '<td width="95px">'; newContent += responseObject.records[i].bank; newContent += '</td>';
 newContent += '<td width="95px">'; newContent += responseObject.records[i].price; newContent += '</td>';
 newContent += '<td width="105px">'; newContent += responseObject.records[i].exchange; newContent += '</td>';
 newContent += '<td width="255px">'; newContent += responseObject.records[i].quoteDateTime; newContent += '</td>';
 newContent += '</tr>'; 
 }
 newContent += '</table>'; 

 document.getElementById('content').innerHTML = newContent;
 }

 </script>

 </head>
 <body>

 <header><h1>360 WebClient </h1></header>
 <br />
 <label>Start Index: </label> <input type= text id="startidx" value="0" >
 <label>Count: </label> <input type= text id="count" value="10" >
 <input id="fetchdata" type="button" value="fetch data" onclick="displayname();" />
 <br />

 <p id="content"></p>
 <br />

 </body>
</html>

AJAX is Asynchronous JavaScript and XML which is how you can call your, or someone else’s web services.  This is neat because you can then create a fairly complicated page but by using AJAX you can pull in other information on demand to create a more interactive web experience.

I am pulling in all of this via the JavaScript library JQuery.  JQuery is not the only JavaScript library but it is one of the most popular ones.  It is possible to get either a compressed or uncompressed version of this library from the JQuery site.

https://jquery.com/download/

My web client allows the user to enter things that interest them and submit that as parameters to the restful web service.  The web service could itself go to the internet, read information from a database or perform some calculations – when it was finished that information could be returned to the calling client.

 myurl = 'http://localhost:8080/ThirdRestful/price/fetch?start=' + start + '&count=' + count;

More specifically, my example queries the values from the user and builds the URI that will call my service with the parameters in their proper locations.

This code is actually very optimistic as it assumes that the user will be smart enough to enter valid values into both fields before pressing the fetch data button.  It also assumes that data will exist.

This last assumption is a bit of a stretch as this is not a bullet proof production program but rather a proof of concept.

Creating test data

Rather than force myself to manually enter the data each time I start my server I have created a restful call to create some data.  This is done by simply calling the following URI.

http://localhost:8080/ThirdRestful/price/generate

This will generate some somewhat random data in order to better test my client. It is also possible to call this method with a parameter to generate more than the default 20 entries. It is possible to call this method any number of times.  The newly generated data will be added to the internal list.

http://localhost:8080/ThirdRestful/price/generate?count=10

The generate method will return the count of the number of items that exist in the list after adding the new day.

This is my last blog article about restful web services for the moment.  This is not the only way to create such services but it is a pretty easy way without having hundreds of additional java classes that make up some frameworks.

I think that this is a superior way of generating web services but any critics could easily point to the fact that it is not easy to know what services exist for my particular application – and they are correct.

Another way of producing web services that allow you to see what services are available is to use WSDL .  This will describe the web services and it also allows you to generate code which can be used to call each of the defined services.

I did have an opportunity to do just this for connecting with SharePoint.  I wasn’t really a big fan of the technology I was given.  There were hundreds of lines of auto-generated code and hundreds of files of odd framework code.  I suspect that some of the other frameworks out there (ie Spring) also have hundreds of new files but probably in a much better organized way.

I may end up checking out one of these other frameworks at some point, but for right now I will enjoy both the power and simplicity of the small Jersey powered restful server technology.

Posted in programming, Setup From Scratch | Tagged , , | Comments Off on Restful services in Java – 360 degree of restful client

Restful services in Java – 360 degree of restful server

This is the blog final entry for a restful service web server.  I have attempted to create more than your typical “foo” “bar” type example program. My final blog on restful services will be a fairly complete service that could be used to provide price data supplied by several “banks” on various exchanges.

The data can be  pseudo randomly generated to be later retrieved and displayed by web browser using java script to fetch, extract and display the data from a JSON formatted string.

A real production server would save server data in a more permanent manner such as a database or even perhaps files if the data lent itself to that. I didn’t feel like creating database so I have skipped over this step by adding a restful service for generating some data which is stored in memory. This is just a “temporary” server that is available for testing and not for real permanence.

This example server has the following methods.

  • generate
  • count
  • (index).xml
  • (index).json
  • (index).text
  • fetch
  • addOne
  • retrieve
  • retrieveall

The generate method is perhaps the most important of these for testing. Simply call this restful service in the beginning to generate twenty items by default or pass in a specific count to generate that many test items.

The generated data is put into a simple hash with the key to the hash being the simple index.

To look at the data that is in the hash the count, (index).xml, (index).json or (index).text.

The fetch method is actually the other main method specifically for a web client. This web service will retrieve an array of market prices in JSON format.

In a few days I will be providing a html client that will use this server and retrieve the data in the JSON format and display that to the web page.

PriceServer.java

package com.acmesoft;

import java.io.File;
import java.io.FileWriter;
import java.io.IOException;

import java.math.BigDecimal;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.HashMap;
import java.util.Iterator;

import javax.ws.rs.Consumes;
import javax.ws.rs.DefaultValue;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.QueryParam;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.UriInfo;
import javax.ws.rs.WebApplicationException;

@Path("/price")
public class PriceServer
{
	// should really be a database
	private static HashMap<Integer,MarketPrice> pricedatabase = new HashMap<Integer,MarketPrice>();
	private static int databasecount = 0;

	//
	// index is the actual 0 - n-1 of the map.
	//
	private String rangeAsXML(int start, int stop)
	{
		String retval = "";
		retval += "";

		for (int idx = start; idx <= stop; idx++)
		{
			MarketPrice item = null;
			item = pricedatabase.get(idx);

			if (item != null)
				retval += item.xmlString();
		}

		retval += "";

		return retval;
	}

	//
	// index is the actual 0 - n-1 of the map.
	//
	private String rangeAsJSON(int start, int stop)
	{
		String retval = "{ \"records\": [";

		for (int idx = start; idx <= stop; idx++)
		{
			MarketPrice item = null;
			item = pricedatabase.get(idx);

			if (item != null)
			{
				if (idx != start)
					retval += ", ";
				retval += item.JSONString();
			}
		}

		retval += "] }";

		return retval;
	}

	// initialize the price cache, this is pretty much only for debugging
	// 
	// this will generate 20 items by default
	// http://localhost:8080/ThirdRestful/price/generate
	//
	// this will generate 100 items instead 
	// http://localhost:8080/ThirdRestful/price/generate?count=100
	@GET
	@Path("/generate")
	@Produces( {MediaType.APPLICATION_XML})
	public Response generate(@Context UriInfo ui, @DefaultValue("20") @QueryParam("count") int debugcount) throws URISyntaxException
	{
		String path = ui.getAbsolutePath().toString() ;
		URI uri = new URI(path);

		MarketPrice newprice = null;
		for (int idx = 0; idx < debugcount; idx++)
		{
			switch (idx % 5)
			{
			case 0:
				newprice = new MarketPrice("EUR/USD", "DBK", new BigDecimal(1.056), "DTB");
				break;
			case 1:
				newprice = new MarketPrice("GBP/EUR", "BCY", new BigDecimal(1.179), "LIFFE");
				break;
			case 2:
				newprice = new MarketPrice("USD/MXN", "C", new BigDecimal(19.1), "CME");
				break;
			case 3:
				newprice = new MarketPrice("EUR/USD", "0UB", new BigDecimal(1.059), "SOFFEX");
				break;
			case 4:
				newprice = new MarketPrice("VEF/USD", "JPM", new BigDecimal(0.1001), "NYSE");
				break;
			}
			newprice.setId(databasecount);
			pricedatabase.put(databasecount, newprice);
			databasecount++;
		}

		//return Response.created(uri).contentLocation(uri).status(Response.Status.OK).build(); 
		return Response.created(uri).contentLocation(uri).status(Response.Status.OK).entity("" + databasecount +"").build(); 
	}

	// get the count of items
	// 
	// http://localhost:8080/ThirdRestful/price/count
	@GET
	@Path("/count")
	@Produces( {MediaType.APPLICATION_XML})
	public Response count(@Context UriInfo ui) throws URISyntaxException
	{
		String retval = "" + databasecount + "";
		String path = ui.getAbsolutePath().toString() ;
		URI uri = new URI(path);

		return Response.created(uri).contentLocation(uri).status(Response.Status.OK).entity(retval).build();
	}


	// add a new price record
	// 
	// http://localhost:8181/ThirdRestful/price/addOne 
	@POST
	@Path("/addOne")
	@Produces( {MediaType.APPLICATION_XML})
	public Response addOne(	@Context UriInfo ui, MarketPrice item) throws URISyntaxException
	{
		String path = ui.getAbsolutePath().toString() ;
		URI uri = new URI(path);
		return Response.created(uri).contentLocation(uri).status(Response.Status.OK).build(); 

	}

	// returns a selection of records in xml format.
	// 
	// http://localhost:8080/ThirdRestful/price/retrieve
	@GET
	@Path("/retrieve")
	@Produces( {MediaType.APPLICATION_XML})
	public Response retrieve(	@Context UriInfo ui,
			@DefaultValue("0") @QueryParam("start") int start, 
			@DefaultValue("10") @QueryParam("count") int count) throws URISyntaxException
	{
		String retval = "";
		String path = ui.getAbsolutePath().toString();
		URI uri = new URI(path);

		if (databasecount == 0)
			return Response.created(uri).contentLocation(uri).status(Response.Status.REQUESTED_RANGE_NOT_SATISFIABLE).build();
		else
		{
			int end = start + count - 1;
			if (end > databasecount-1)
				return Response.created(uri).contentLocation(uri).status(Response.Status.REQUESTED_RANGE_NOT_SATISFIABLE).build();
			retval = rangeAsXML(start,end);
		}
		return Response.created(uri).contentLocation(uri).status(Response.Status.OK).entity(retval).build(); 
	}

	// returns a selection of records in xml format.
	// 
	// this will return 10 items starting with the first
	// http://localhost:8080/ThirdRestful/price/fetch
	//
	// start with index one for three items
	// http://localhost:8080/ThirdRestful/price/fetch?start=1&count=3

	@GET
	@Path("/fetch")
	@Produces( {MediaType.APPLICATION_JSON})
	public Response fetch(	@Context UriInfo ui,
			@DefaultValue("0") @QueryParam("start") int start, 
			@DefaultValue("10") @QueryParam("count") int count) throws URISyntaxException
	{
		String retval = "";
		String path = ui.getAbsolutePath().toString();
		URI uri = new URI(path);

		if (databasecount == 0)
			return Response.created(uri).contentLocation(uri).status(Response.Status.REQUESTED_RANGE_NOT_SATISFIABLE).build();
		else
		{
			int end = start + count - 1;
			if (end > databasecount-1)
				return Response.created(uri).contentLocation(uri).status(Response.Status.REQUESTED_RANGE_NOT_SATISFIABLE).build();
			retval = rangeAsJSON(start,end);
		}
		return Response.created(uri).contentLocation(uri).status(Response.Status.OK).entity(retval).build(); 
	}


	// returns all records in xml format.
	// 
	// http://localhost:8080/ThirdRestful/price/retrieveall 
	@GET
	@Path("/retrieveall")
	@Produces( {MediaType.APPLICATION_XML})
	public Response showall(@Context UriInfo ui) throws URISyntaxException
	{
		String retval = "";
		String path = ui.getAbsolutePath().toString() ;
		URI uri = new URI(path);

		retval = rangeAsXML(0,databasecount-1);
		return Response.created(uri).contentLocation(uri).status(Response.Status.OK).entity(retval).build();
	}



	// returns single record in xml format.
	// 
	// http://localhost:8080/ThirdRestful/price/34.xml
	@GET
	@Path("{idx}.xml")
	@Produces( {MediaType.APPLICATION_XML})
	public Response getIndexXml(@Context UriInfo ui, @PathParam("idx") int idx)  throws URISyntaxException
	{
		MarketPrice retval = null;
		String path = ui.getAbsolutePath().toString() ;
		URI uri = new URI(path);

		if (idx >= 0 && idx < databasecount) 
                { 
                        retval = pricedatabase.get(idx); 
                        return Response.created(uri).contentLocation(uri).status(Response.Status.OK).entity(retval).build(); 
                } 
                else 
                        return Response.created(uri).contentLocation(uri).status(Response.Status.REQUESTED_RANGE_NOT_SATISFIABLE).build(); 
        } 

        // returns single record in json format. 
        // 
        // http://localhost:8080/ThirdRestful/price/34.json 
        @GET @Path("{idx}.json") 
        @Produces( {MediaType.APPLICATION_JSON}) 
        public Response getIndexJSON(@Context UriInfo ui, @PathParam("idx") int idx) throws URISyntaxException 
        { 
                MarketPrice item = null; 
                String retval = ""; 
                String path = ui.getAbsolutePath().toString() ; 
                URI uri = new URI(path); 
                if (idx >= 0 && idx < databasecount) 
                { 
                        item = pricedatabase.get(idx); 
                        retval = item.JSONString(); 
                        return Response.created(uri).contentLocation(uri).status(Response.Status.OK).entity(retval).build(); 
                } 
                else 
                        return Response.created(uri).contentLocation(uri).status(Response.Status.REQUESTED_RANGE_NOT_SATISFIABLE).build(); 
         } 

         // returns single record in json format. 
         // 
         // http://localhost:8080/ThirdRestful/price/34.text 
         @GET @Path("{idx}.text") 
         @Produces( {MediaType.TEXT_PLAIN}) 
         public Response getIndexTEXT(@Context UriInfo ui, @PathParam("idx") int idx) throws URISyntaxException 
         {
                MarketPrice item = null; 
                String retval = ""; 
                String path = ui.getAbsolutePath().toString() ; 
                URI uri = new URI(path); 
                if (idx >= 0 && idx < databasecount)
		{
			item = pricedatabase.get(idx);
			retval = item.toString();
			return Response.created(uri).contentLocation(uri).status(Response.Status.OK).entity(retval).build();
		}
		else
			return Response.created(uri).contentLocation(uri).status(Response.Status.REQUESTED_RANGE_NOT_SATISFIABLE).build();
	}


}

MarketPrice.java

package com.acmesoft;

import java.io.StringWriter;
import java.math.BigDecimal;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;

import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Marshaller;
import javax.xml.bind.PropertyException;
import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlRootElement;

import com.google.gson.Gson;

import javax.xml.bind.annotation.XmlAccessType;;

@XmlRootElement(name="marketPrice")
@XmlAccessorType(XmlAccessType.FIELD)
public class MarketPrice 
{
	@XmlElement
	private int id;
	@XmlElement
	private String symbol;		// EUR/USD
	@XmlElement
	private String bank;		// DBK
	@XmlElement
	private BigDecimal price;	// 1.1222
	@XmlElement
	private String exchange;	// DTB, NYSE, CBOT
	@XmlElement
	private Date quoteDateTime;	// 2015-01-01 14:32:45.032


	public MarketPrice()
	{
		id = -1;
		this.symbol= "";
		this.bank = "";
		this.price = new BigDecimal(0.0);
		this.exchange = "";
		this.quoteDateTime = new Date();
	}


	public MarketPrice(String symbol, String bank, BigDecimal price, String exchange)
	{
		DateFormat dateFormat = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss");

		this.symbol= symbol;
		this.bank = bank;
		this.price = price;
		this.exchange = exchange;
		this.quoteDateTime = new Date();
		String debugDate = dateFormat.format(quoteDateTime);

		id = -1;
	}

	public MarketPrice(String symbol, String bank, BigDecimal price, String exchange, Date quoteDateTime)
	{
		this.symbol= symbol;
		this.bank = bank;
		this.price = price;
		this.exchange = exchange;
		this.quoteDateTime = quoteDateTime;
		id = -1;
	}

	public void setId(int value)
	{
		id = value;
	}
	public int getId()
	{
		return id;
	}

	public void setSymbol(String value)
	{
		symbol = value;
	}
	public String getSymbol()
	{
		return symbol;
	}

	public void setBank(String value)
	{
		bank = value;
	}
	public String getBank()
	{
		return bank;
	}

	public void setPrice(BigDecimal value)
	{
		price = value;
	}
	public void setPrice(double value)
	{
		price = new BigDecimal(value);
	}
	public BigDecimal getPrice()
	{
		return price;
	}

	public void setExchange(String value)
	{
		exchange = value;
	}
	public String getExchange()
	{
		return exchange;
	}

	public Date getQuoteDateTime()
	{
		return quoteDateTime;
	}

	public void setQuoteDateTime(Date value)
	{
		quoteDateTime = value;
	}


	public String toString()
	{
		String retval = getCSVData();

		return retval;
	}

	public String xmlString()
	{ 
		String retval = toXmlString();
		retval = retval.substring(retval.indexOf("\n"));

		return retval;
	}

	public String JSONString()
	{ 
		Gson gson = new Gson();
		String JSONstr = gson.toJson(this);
		return JSONstr;  
	}

	public String toXmlString()
	{
		JAXBContext ctx = null;
		Marshaller maxMarshaller = null;

		try {
			ctx = JAXBContext.newInstance(MarketPrice.class);
		} catch (JAXBException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}

		try {
			maxMarshaller = ctx.createMarshaller();
		} catch (JAXBException e1) {
			// TODO Auto-generated catch block
			e1.printStackTrace();
		}

		try {
			maxMarshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true);
		} catch (PropertyException e1) {
			// TODO Auto-generated catch block
			e1.printStackTrace();
		}

		StringWriter writer = new StringWriter();
		try {
			maxMarshaller.marshal(this,writer);
		} catch (JAXBException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}

		String xmlPrice = writer.toString();

		return xmlPrice;
	}

	private String convertToTime(long milliseconds)
	{
		Date date = new Date(milliseconds);
		DateFormat dateformat = new SimpleDateFormat("HH:mm:ss.SSS");
		return dateformat.format(date);
	}
	public String getCSVHeader()
	{
		String delim = com.acmesoft.Constants.FIELDDELIM;
		String retval;

		retval = "Id" + delim + "Symbol" + delim + "Price" + delim + "Bank" + delim + "Exchange" + delim + "QuoteDate" ;

		return retval;
	}
	public String getCSVData()
	{
		String delim = com.acmesoft.Constants.FIELDDELIM;
		String retval;

		retval = id + delim + symbol + delim + price + delim + bank + delim + exchange + delim + quoteDateTime ;

		return retval;
	}
}

PriceCount.java

package com.acmesoft;

import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlRootElement;

@XmlRootElement(name="marketPrice")
@XmlAccessorType(XmlAccessType.FIELD)
public class PriceCount
{
        @XmlElement
        private int count;

        public PriceCount()
        {

        }
        public PriceCount(int val)
        {
                count = val;
        }

        public int getCount()
        {
                return count;
        }

        public void setCount(int val)
        {
                count = val;
        }
}

Constants.java

package com.acmesoft;

public class Constants {
	final public static String FIELDDELIM = ",";
	final public static int OK = 200;
	final public static String DATETIMEFORMAT = "yyyy-MM-ddHH:mm:ss.SSS";
}
Posted in programming | Tagged , , | Comments Off on Restful services in Java – 360 degree of restful server

command line fun – unlimited command line length

While looking into another problem, plethora of shells, I was initially afraid that some of my fooling around was causing some odd side effects.  While investigating I ended up with a different solution that was actually overkill but it help eliminate other options as a source of the problem.

My solution was to create a file with the names of all of the files that I needed to process and use that as my input.   This allowed me to look at as many files as I wanted to without seeing anything like this.

-bash: /usr/bin/ls: Arg list too long

This isn’t the only solution, nor even the most efficient but it sure is a fun one that uses both head and tail on the same line.  A more efficient solution would probably to use xargs, find or perhaps a more efficient shell script.

Rather than to do the efficient, I am going to show off the fun solution.  This is something you could use even in a production environment as long as the number of files is reasonably small.

I decided to use both head and tail to pick out lines from a file for processing.  Simply save a list of all files to a temporary file and then process each file one at a time.

The fun example

INPUT=/home/bob/input
TMPFILE=`mktemp /tmp/XXXXXX`

pushd `pwd`
cd $INPUT
ls -1 > $TMPFILE
CNT=`cat $TMPFILE | wc -l`
IDX=1
while [ $IDX -le $CNT ]
do
   NAME=`head -n ${IDX} $TMPFILE | tail -1`

   echo processing $NAME
   # do some processing here

   IDX=$(($IDX +1)) 
done
popd
rm $TMPFILE

The actual interesting part is where the variable NAME is being assigned but before getting ahead of myself, here is a blow by blow explanation of the script.

# Line Description
1. INPUT=/home/bob/input set up our input location
2. TMPFILE=`mktemp /tmp/XXXXXX` create a unique filename in the /tmp directory
3. pushd `pwd` save our current directory
4. cd $INPUT change to input directory
5. ls -1 > $TMPFILE save a list of file names (no directory) into our temporary file.
6. CNT=`cat $TMPFILE | wc -l` get a count of the number of lines in our file.
7. IDX=1 which line of the file we are on at the moment.
8. while [ $IDX -le $CNT ] while there are lines left to process
9. do
10. NAME=`head -n ${IDX} $TMPFILE | tail -1` get the entry from the file.
11. echo processing $NAME in this case echo the filename to the screen.
12. # do some processing here nothing to see here, move along.
13. IDX=$(($IDX +1)) increase index to the next line
14. done
15. popd change back to original directory
16. rm $TMPFILE remove temporary file.

The code is pretty obvious what it is doing for most of the script.  The interesting lines might be 2 and 10.

The mktemp command simply creates a unique file that is six characters long designated by the XXXXXX format.  This is really useful as the operating system will do the hard work for you.  It is even possible to create a subdirectory with a file or just a unique subdirectory.

Line number ten is actually pretty simple when you think about it. The head command will take that many lines from our temporary file.  the first time through this will be the first line.  the tail command will then take the last line from this batch.  When the index is 1 both the head and tail will deliver a single line.

When the index is larger (ie 5) then more lines will be delivered.  The head command will deliver all of the previously processed lines along with one new one.  The tail command will pull off that one new line so it can be processed.

This method of processing data might also be interesting if there are spaces in the filenames or paths.  You would know with certainty that each line represents a single parameter or value.

Again, this is a terribly inefficient way of processing through a million lines, especially if these lines are each hundreds of characters long.

Posted in Command line | Tagged , | Comments Off on command line fun – unlimited command line length

Gentlemen’s vandal

We were watching the BBC world news a few days back and it must have been a slow news day. The reporter was speaking with someone whose face was hidden. It turned out that this particular vigilante defaced buildings and has been doing so for the past 13 years.

No, he wasn’t putting up gang graffiti letters nor faces, animals or general art. His changes were very very tiny. So tiny that the actual shop owners in some cases didn’t recognize anything was different. So what was the change?

He would either add or remove apostrophe’s as required.

Ie.

Amys Nail’s -> Amys Nails

The interviewee never said what his profession is but it could easily be that of an English teacher considering how my teacher was during English class.

It is hard to encourage vandalism but this particular type is at least a refreshing change from the norm.

A “grammar vigilante” sneaks around at night fixing an infuriatingly common error on public signs

Posted in Soapbox | Comments Off on Gentlemen’s vandal

Restful services in Java – client and server

My previous restful client example and server example were perfect, well, not exactly.  They worked but they pretty much assuming a perfect world.  That is to say that there is never a communication error or a server error.

This small example will return one of the magazine objects.  Depending how you call it, it will return either the object directly or the xml that you can marshal into the object itself.

// example 2/3 setup
ClientConfig config1 = new ClientConfig();
Client client1       = ClientBuilder.newClient(config1);
WebTarget target1    = client1.target(uribase);
		
// example 3
magazine xmlobj = target1.path("/Communicate/hello").request().accept(MediaType.APPLICATION_XML).get(magazine.class);
showMagazine("pure xml object", xmlobj);

However, it should be pretty obvious in the previous simple server code that if for any reason the server cannot find the data it cannot really communicate that fact back to the client.  Well, it cannot if you are returning magazine objects or general strings.

New and better server

The solution is to return a response record.  This record will contain both the error code as well as our object.  The code is actually not too different from from the previous example but there are some major differences.

The first is that we are no longer returning either xml(string) nor magazine objects but rather we are returning Response objects.    This allows us to set the status to one of the values that matches the result of the call.

Response.Status.OK

It is possible to use one of the predefined values such as Response.Status.OK but it is just as acceptable to put the actual value of 200.

Here is a list of the values that are defined for us to use.

Response codes Description
OK 200
CREATED 201
ACCEPTED 202
NO_CONTENT 204
MOVED_PERMANENTLY 301
SEE_OTHER 303
NOT_MODIFIED 304
TEMPORARY_REDIRECT 307
BAD_REQUEST 400
UNAUTHORIZED 401
FORBIDDEN 403
NOT_FOUND 404
NOT_ACCEPTABLE 406
CONFLICT 409
GONE 410
PRECONDITION_FAILED 412
UNSUPPORTED_MEDIA_TYPE 415
INTERNAL_SERVER_ERROR 500
SERVICE_UNAVAILABLE 503

The code in this “server code” is not much larger than the original code.

Another one of the changes is that the URI is returned as part of the response object.  This URI should point to the the data in the server.  This is especially interesting if a new object is created on your server.  The server then will return the URI to reference that object.

The final difference is that we pass back our object as part of the Response object.  In the below code this is done as a simple string but it could just as easily be a your own Java object.

package com.acmesoft;

import java.net.URI;
import java.net.URISyntaxException;

import javax.ws.rs.Consumes;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.UriInfo;

@Path("/Communicate")
public class HelloWorldEx {

	/*  
	 * http://192.168.178.70:8080/SecondRestful/Communicate/mag
	 */	@GET
	@Path("/mag")
	@Produces( {MediaType.APPLICATION_XML})
	public Response sayHelloXml(@Context UriInfo ui) throws URISyntaxException
	{
		String retval =  "" +
				"2016" +
				"6" +
				"Economist" +
				"9.99" +
				"";
		String path = ui.getAbsolutePath().toString() ;
		URI uri = new URI(path);

		return Response.created(uri).contentLocation(uri).status(Response.Status.OK).entity(retval).build();
	}

	/*  
	 * http://192.168.178.70:8080/SecondRestful/Communicate/mag
	 */
	@GET
	@Path("/mag")
	@Produces( {MediaType.APPLICATION_JSON})
	public Response sayHelloJSON(@Context UriInfo ui) throws URISyntaxException
	{
		String retval =  "{ \"magazine\": {\"year\":2016,\"month\":6,\"title\":\"Economist\",\"price\":9.99} }";
		
		String path = ui.getAbsolutePath().toString() ;
		URI uri = new URI(path);

		return Response.created(uri).contentLocation(uri).status(Response.Status.OK).entity(retval).build();
	}

	/*  
	 * http://192.168.178.70:8080/SecondRestful/Communicate/oucha_bunga
	 */
	@Path("{c}")
	@GET
	@Produces( {MediaType.APPLICATION_XML})
	public Response complexURI(@Context UriInfo ui, @PathParam("c") String msg) throws URISyntaxException
	{
		String mymsg = "Thinkin about " + msg;
		String xmlStr = "" + mymsg + "";

		// this is a really clever way to get the URI but
		// if you pass in a string, it will throw an error.
		// and it all comes down to the URI that is added to the
		// response object (below)
		String path = ui.getAbsolutePath().toString() ;
		URI uri = new URI(path);

		return Response.created(uri).contentLocation(uri).status(Response.Status.OK).entity(xmlStr).build();
	}
}

The final method in this server example is is not actually called by my client but it is a clever example of a URI that can accept anything, in this case a string, and then that string can be used in some clever manner.

New Client

The new client changes is actually quite similar to the new server changes.  As a matter of fact, the client changes are actually almost transparent. There is just a few small changes to accommodate the retrieving Response objects instead of strings or other java objects.

Once the Response object has been retrieved simply call the status method to find out the status returned from the server. The range of success codes is 200-299.

	if (response.getStatus() == 200)
	{
		xmlAnswer = response.readEntity(String.class);
		System.out.println(xmlAnswer);
		magazine converted = convertXmlToMag(xmlAnswer);
		showMagazine("xml text data", converted);			
	}
	response.close();

If the call was successful then simply call the readEntity method on the Response object.  Simply pass in the class of the object that you are trying to read out.  The object is retrieved just like magic.

In order to keep control on the system resources close the Response and the Client objects, failing to do so will cause your program to fail eventually due to resource issues.

package com.acmesoft;

import java.io.StringReader;
import java.net.URI;

import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.client.WebTarget;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.UriBuilder;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Unmarshaller;

import org.glassfish.jersey.client.ClientConfig;

public class TestClient 
{
	String uribase = "http://192.168.178.70:8080/SecondRestful";

	public void showMagazine(String desc, magazine value)
	{
		System.out.println("\n" + desc );
		System.out.println("year "  + value.getYear());
		System.out.println("month " + value.getMonth());
		System.out.println("price " + value.getPrice());
		System.out.println("title " + value.getTitle());
	}
	
	public magazine convertXmlToMag(String xmlstring)
	{
		magazine converted = null;
		JAXBContext jc = null;
		try {
			jc = JAXBContext.newInstance(magazine.class);
			Unmarshaller unmarshaller = jc.createUnmarshaller();
			StringReader reader = new StringReader(xmlstring);
			converted = (magazine) unmarshaller.unmarshal(reader);	
		}
		catch (JAXBException ex)
		{
			ex.printStackTrace();
		}
		return converted;
	}
	
	public WebTarget restConnect(String targetUrl)
	{
		URI uri = UriBuilder.fromUri(targetUrl).build();
		ClientConfig config = new ClientConfig();
		Client client = ClientBuilder.newClient(config);
		WebTarget target = client.target(uri);
		return target;
	}
	
	public TestClient()
	{
		ClientConfig config = new ClientConfig();
		Client client       = ClientBuilder.newClient(config);
		WebTarget target    = client.target(uribase);

		Response response  = target.path("/").path("Communicate").path("/mag").request().accept(MediaType.APPLICATION_XML).get(Response.class);
		System.out.println(response.getStatus());
		String xmlAnswer = "";
		if (response.getStatus() == 200)
		{
			xmlAnswer = response.readEntity(String.class);
			System.out.println(xmlAnswer);
			magazine converted = convertXmlToMag(xmlAnswer);
			showMagazine("xml text data", converted);			
		}
		response.close();

		response = target.path("/Communicate/mag").request().accept(MediaType.APPLICATION_XML).get(Response.class);
		if (response.getStatus() == 200)
		{
			magazine xmlobj = response.readEntity(magazine.class);
			showMagazine("pure xml object", xmlobj);
		}
		response.close();
		client.close();
	}
	
	public static void main(String[] args) 
	{
		new TestClient();
	}
}

The output is not terribly interesting, it is just a few print statements in order to prove that the client can retrieve the data.

 200
<Magazine><Year>2016</Year><Month>6</Month><Title>Economist</Title><Price>9.99</Price></Magazine>

xml text data
year 2016
month 6
price 9.99
title Economist

pure xml object
year 2016
month 6
price 9.99
title Economist

With the ability to verify the return status of the restful server calls it is possible to create much more interesting services than displayed here.

I have attempted to create more than your typical “foo” “bar” type example program. My final blog on restful services will be a fairly complete service that could be used to provide price data supplied by several “banks” on various exchanges.

The data is actually pseudo randomly generated and retrieved and displayed by web browser using java script to fetch, extract and display the data from a JSON formatted string.

Posted in programming, Setup From Scratch | Tagged , , | Comments Off on Restful services in Java – client and server