Paranoid Professor | Technology, security and the news for the rest of us.

paranoid about printing

Posted on October 12, 2016 by DocDock

It seems that the venerable Hewlett Packard has been in the news recently about ink jet cartridges[arstechnia][forbesadvocate]. Allegedly HP changed their firmware to favor their own ink while preventing other ink cartridges from working. This apparently also includes refilled genuine ink cartridges.

The substance of the HP firmware change came out when at some point the printer would report that the non-HP cartridges were damaged. Oddly enough, this occurred for most of those printer owners on September 13th.

This doesn’t affect me as I chisel all of my code listings into small clay tablets when I am not using a laser printer.

After all of this coverage, and coincidentally have to replace my toner cartridge last week I thought I would check a bit more into this. The good news is that the internet doesn’t have much to say about toner cartridges having particular limiting factors preventing full use of them. Less good news is that I did find a young girl who is playing with toner like it was flour. The information on the internet is not that toner itself is so bad for you as the size of the particles make it easy for them to get into your cells and that might not be so good for you.

However after watching this young girl play with HP laser toner cartridges I begane to wonder just how empty my toner cartridges really are. After all, it seemed to be a fair amount of toner and most likely this girl’s parents wouldn’t have given her a new cartridge to play with.

I am not ready to start clipping wires or covering sensors on my printer but I am quite capable of going into the printer configuration menu. It seems to me that the settings that were selected, probably by default, were perhaps a bit conservative. I personally would like my printer to print until I detect that the pages come out badly due to lack of toner. I also doubt that most of my pages are as densely packed as HP might think so I did a few small changes.

There really wasn’t much to choose from, however, I was able to change the threshold of when the printer should stop printing.

I don’t have a manual handy but I am guessing that this 1 – 20 range is how many pages of toner is left before it stops printing. I would think that six pages is silly, if enough toner exists, then the printer should generate until only one page is left.

hp-replace-supplies

I am speculating at this point that this option will allow me to override the printer’s desire to stop printing pages even when it thinks there is no more toner. If indeed there is no more toner, I will replace it, otherwise, lets give it a chance to prove it is out. So I change this to “Override out”.

I did make one other unrelated discovery while checking out my printer configurations. There is also a potential power saving to be had just by changing how long the printer will remain awake after printing. Changed this to 1 minute.

I do understand why any company would provide their product a bit cheaper or even at a loss with the hopes of locking in their customer into a long relationship of consumables. After all this strategy did work well for razors and undoubtedly for other companies as well.

It isn’t always the case, but this story does end on a happy note. In this day and age the internet can get information out in front of millions of eyeballs in literally seconds. This is good news for man saves kitten from highway but less advantageous if the story is about other poor behavior that you might wish to keep out of sight.

HP apparently had the opportunity to consider the impact of their “safety” firmware release and have decided to undo that particular bit of logic.

Extreme measures

Most of the documents that get printed out from my computer are not meant to be a work of art – normally it is a quick print out to verify a point or as a rough draft to be examined by someone else.

There are a number of people on the internet that believe that the amount of free toner is reduced for all colors even for printouts that are mainly in black and white printouts. I cannot prove or disprove this but I have decided to do a small experiment. I have created a second copy of my printer on my computer and set it up to only print in black and white.

Forms from other companies that I need to print out may no longer have their colorful logos but that is fine by me. If I need to print in color, just use the second defined printer which can print in color.

( I thought it would be as easy as a few clicks printer setup, but in the end I had to create a second port and do some other setup. )

More Extreme measures

Normally this would have been the end of the story, but now I am really interested in printer toner cartridges. While writing this all down I also checked into how to re-fill your toner cartridges. I wasn’t so surprised to find a long video on how to do this process on youtube.

The refill kits are actually a lot cheaper than purchasing vendor toner cartridges, but I guess it depends on how hands on your wish to be when dealing with your printer.

I did a small survey among some of my tech friends and most of them decided that it wasn’t worth their effort to manually fill their cartridges and for their needs they were content with the vendor toner cartridges.

Posted in Soapbox | Tagged paranoid | 1 Comment

compiling and making jar files with Ant

Posted on October 8, 2016 by DocDock

When I first saw Java I was surprised at exactly how embarrassingly simple it was to compile a program. Well, with the caveat that the file or files are all in the same directory.

javac *.java

When the number of source files is tiny and the number of other files is equally small a simple directory with two or three files seems like an ok solution – even for production. It turns out that wasn’t really a ok solution. The fact was my previous boss had set the bar pretty low for what was deemed to be acceptable.

When the standards went up, I had to up my game a bit. One of the easier ways to do this is to create java libraries instead of just having a loose confederation of files.

Manual jar creation

A jar file is actually just a simple archive of files in the given directory with the addition of a manifest file. Manually creating a jar file can be done with the following command.

jar cvfm hello.jar manifest.mf de/

The manifest file actually contains some information about the contents of the jar file.

Manifest-Version: 1.0
Built-By: chdock
Main-Class: de.companyname.helloworld.HelloWorld
Implementation-Version: 3.15.047 build #6
Built-Date: 2016-09-01 10:01:14

As you can see most of this is version information, who built it and when. The “Main-Class” file field actually points to the main program in the jar file. This entry allows the jar file to be run directly without giving the name of the program. This does assume that no other external jar files are required.

That isn’t to say that you cannot have an runnable jar when external jar files are necessary. When that happens simply add them to the classpath in the manifest file.

Class-Path: ..\lib\log4j-core-2.3.jar ..\lib\log4j-api-2.3.jar ..\resources\log4j2.xml

The classpath is relative to the location of the jar file you are running.

Running program

Running the jar directly is done using the -jar argument.

java -jar yourlibraryname.jar

Of course this doesn’t actually prevent running the program or other programs directly from the jar in the normal manner.

java -cp lib\lib1.jar;lib\lib2.jar;.\yourlibraryname.jar ProgramNameHere

Of course I could have written a shell script to take the jar command to create the libraries, but rather than try and force jar and bash shell together I opted for a more proper solution which was to use Apache Ant instead.

Apache Ant is somewhat similar to the Unix make command. You define targets that need to be performed. When creating each of these targets you can make them dependent on another target. However, the actual order of the execution is not set other than to obey the dependencies.

Below is a sample Ant file to compile the source code and create a jar file once that is done. The default name for the ant configuration file is called “build.xml”, however, it is possible to run Ant with a build file of any name. This is done by passing the name in on the command line when starting ant.

Simple build file

build.xml

<project default="world" basedir=".">

        <property name="version.num" value="3.15.047"/>
        <property name="build.num" value="6"/>
        <property name="mainclass" value="de.companyname.helloworld.HelloWorld"/>
        <property name="lib.dir" value="${basedir}/libs"/>
        <property name="src.dir" value="${basedir}/src"/>
        <property name="build.dir" value="${basedir}/bin"/>
        <property name="jarname" value="hello"/>
        <property name="jar.dir" value="${basedir}/final"/>

        <path id="classpath">
                <fileset dir="${lib.dir}" includes="**/*.jar"/>
        </path>
 
        <target name="compile" description="compile source code">
                <echo>"compiling ..."</echo>
                <mkdir dir="${build.dir}"/>
                <javac srcdir="${src.dir}" destdir="${build.dir}" classpathref="classpath"/>
        </target>
 
        <target name="jarfile" depends="compile" description="generate jar file">
                <echo>"build jar ..."</echo>
                <mkdir dir="${jar.dir}"/>
 
                <tstamp>
                        <format property="today.timestamp" pattern="yyyy-MM-dd HH:mm:ss" />
                </tstamp>
 
                <jar destfile="${jar.dir}/${jarname}.jar" basedir="${build.dir}">
                <manifest>
                 <attribute name="Built-By" value="${user.name}" />
                 <attribute name="Main-Class" value="${mainclass}" />
                 <attribute name="Implementation-Version" value="${version.num} build #${build.num}"/>
                 <attribute name="Built-Date" value="${today.timestamp}"/>
                </manifest>
                </jar>
        </target>
 
        <target name="world" depends="jarfile">
                 <echo>"build script version 1.00"</echo>
        </target>
</project>

This particular Ant script is probably one of the smallest which will compile our code and also create a runnable jar file. It could be made a bit smaller by removing the properties at the top of the script and embedding those values where they are used.

This script can either be used by Eclipse to build your programs or using Ant from the command prompt.

> ant world
Buildfile: /tmp/simplejar/build.xml                                                                                                                       
                                                                                                                                                          
compile:                                                                                                                                                  
     echo "compiling ..."                                                                                                                               
    javac /tmp/simplejar/build.xml:19: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
    javac Compiling 1 source file to /tmp/simplejar/bin

jarfile:
     echo "build jar ..."
    mkdir Created dir: /tmp/simplejar/final
      jar Building jar: /tmp/simplejar/final/hello.jar

world:
     echo "build script version 1.00"

BUILD SUCCESSFUL
Total time: 3 seconds

Dealing with the source code

Once we have a working Ant script it is possible to make small changes here and there to create some nice additional effects. To simplify the build script, I have moved some of the properties into a small file called project.properties which is then included.

project.properties

version.num = 3.13.047
build.num = 9
mainclass = de.companyname.helloworld.HelloWorld
jarname = communication
zipfile = communication-src
#copysource = true

build.xml

<project default="world" basedir=".">

        <property file="project.properties"/>

        <property name="mainclass" value="en.companyname.helloworld.HelloWorld"/>
        <property name="lib.dir" value="${basedir}/libs"/>
        <property name="src.dir" value="${basedir}/src"/>
        <property name="build.dir" value="${basedir}/bin"/>
        <property name="jarname" value="hello"/>
        <property name="jar.dir" value="${basedir}/final"/>

        <path id="classpath">
                <fileset dir="${lib.dir}" includes="**/*.jar"/>
        </path>

        <target name="copysrc" description="copy source code" if="copysource">
                <copy todir="${build.dir}">
                        <fileset dir="${src.dir}" includes="**/*.java"/>
                </copy>
        </target>

        <target name="compile" description="compile source code">
                <echo>"compiling ..."</echo>
                <mkdir dir="${build.dir}"/>
                <javac srcdir="${src.dir}" destdir="${build.dir}" classpathref="classpath"/>
        </target>

        <target name="jarfile" depends="compile" description="generate jar file">
                <echo>"build jar ..."</echo>
                <mkdir dir="${jar.dir}"/>

                <tstamp>
                        <format property="today.timestamp" pattern="yyyy-MM-dd HH:mm:ss" />
                </tstamp>

                <jar destfile="${jar.dir}/${jarname}.jar" basedir="${build.dir}">
                <manifest>
                        <attribute name="Built-By" value="${user.name}" />
                        <attribute name="Main-Class" value="${mainclass}" />
                        <attribute name="Implementation-Version" value="${version.num} build #${build.num}"/>
                        <attribute name="Built-Date" value="${today.timestamp}"/>
                </manifest>
                </jar>
        </target>

        <target name="clean" description="clean up after ourselves">
                <echo>"cleaning ..."</echo>

                <mkdir dir="${build.dir}"/>
                <delete dir="${build.dir}"/>

                <mkdir dir="${jar.dir}"/>
                <delete dir="${jar.dir}"/>

                <mkdir dir="${basedir}/${jarname}" />
                <delete dir="${basedir}/${jarname}" />
        </target>

        <target name="package" depends="compile" description="make a source zip file ">
                <echo>"packaging ..."</echo>
                <mkdir dir="${basedir}/${jarname}/${jarname}-v${version.num}" />

                <copy todir="${basedir}/${jarname}/${jarname}-v${version.num}">
                        <fileset dir="${src.dir}" includes="**/*.java"/>
                </copy>

                <zip destfile="${jar.dir}/${jarname}-v${version.num}.zip" basedir="${basedir}/${jarname}/${jarname}-v${version.num}"/>
        </target>

        <target name="world" depends="jarfile">
                <echo>"build script version 1.00"</echo>
        </target>
</project>

This build file is not significantly more complex than the simple example. The main difference is it can also create a zip file containing the source code, or it can even place the source code into the jar file itself.

Why would you want to put the source into the jar file? Well, this would be mainly for dealing with the situation of someone who is fairly thick and who cannot be trusted to keep both the program and the source code in sync. When it is done in this manner then the person who has the program also has the source by default. This is probably not the best idea depending on the size of the codebase – but that is a discussion for another time.

This build script can also create a zip file containing the source code is created at the same time as the jar file and placed into the final directory.

How to enable putting the source code into the jar file? Simply remove the hash symbol from in front of the property “copysource” in the properties file. This will cause the copysource property to be set and thus cause the copy source target to be invoked. Invoked how? Simple, this target has one additional parameter if=”copysource” added to it. This property acts as a Boolean that will execute the target only when it is set.

For anyone with a really sharp eye for detail will notice that the properties jarname and mainclass exist in both the project.properties file and as properties in the build script. Curiously both are different values. How does Ant decide which value to use? It is actually quite easy to see what values will be used while checking through the script.

Once Ant assigns a value to a property it cannot be changed. So, in this case the values from the project.properties file will be assigned first and thus the values in the build script will be ignored (for jarname and mainclass) as they are already defined.

A more common configuration would be to have a property file for the program (ie default.properties) and to have another property file (build.properties) that could be created to alter from the default behavior. Both files would be read into the build script and if the developer wished to use some slightly different values in the build (ie turn on debugging) it could be done at that point.

Below is my awesome “hello world” application in java as well as how to run it directly from it’s jar file.

HelloWorld.java

 
package de.companyname.helloworld; 
public class HelloWorld 
{ 
        public static void main(String args[]) 
        { 
                System.out.println("hello world"); 
        } 
}

runInterface.sh

#!/bin/bash  

CMD="java -jar final/hello.jar"
echo $CMD
$CMD

If you have liked this blog entry about Ant you might also like the follow up where I do more than just build a jar file.

Note: The example was compiled with java 1.7.

Posted in programming | Tagged best practices, java | Comments Off

a reflection of current events – brexit

Posted on October 2, 2016 by DocDock

I was visiting a local high school when I learned that children can be so cruel.

what brexit looks like

Posted in Soapbox | Tagged UK | Comments Off

Never ever rush … software installation

Posted on September 30, 2016 by DocDock

I should have known better but how hard could the task really be? Download some free or open source software from the internet, configure it, and presto I then have all of the tools necessary for my “homework” assignment.

The task was to create a small restful server and a small client application to make a number of calls to this server. I think that one of the other preconditions was that the resulting program also used something from the SDK installation. I really cannot remember all of the details other than don’t rush.

I wanted to get the program done in the next week and had some quiet time on Sunday to get everything installed and setup. Instead of getting a working environment I ended up getting a crapware installed environment that was doing an amazing job of hijacking all of my web browsers.

I was lucky that my friend Mikhail was able to ~~decrap~~ remove the malware so I didn’t need to reinstall windows. I didn’t want to reinstall windows as the big M makes some simplifying assumptions about how many operating systems will be on the newly installed computer.

Lets be honest, I didn’t pay enough attention when installing each of these packages. Who hasn’t downloaded and installed some viewer or other tool to see that the vendor has teamed up with a anti-virus company. If not that then some vendor also includes a must have “tool bar” for your browser despite the fact I have gotten along quite nicely without their offering.

It must have been one of those types of packages that installed this extra special payload. I downloaded both software from both very respectable firms as well as a download site or two. I think that mainly that is where by problems began.

software bundles

Nobody with an ounce of business acumen will advertise that they are providing goods or services that you don’t want and probably would never want. Not only don’t you want it but the odds are it will have negative effects. I guess software bundles sounds much nicer.

This isn’t really new, computer companies have been bundling their own software, utilities and other third party software. The companies call it bundling while some of the users and media call that particular solution bloatware [0]. I have always wondered why companies would exchange a few dollars per machine when eventually what they were doing would give them a negative image.

This type of behavior has taken some of the fun out of computers. Now you need to have virus checkers and malware checkers just to turn on the computer. It might be a bit paranoid but testing out unknown software should probably be now be done in a virtual machine.

The same would be true when receiving any emails with attachments from people you don’t know. This can be true for people you do know but where the content appears suspicious compared to previous correspondence (sorry mom).

This level of paranoia is certainly not within the ability/effort of the casual computer person and so the bundler’s win.

The solution

In the situation where download sites are injecting extra software [1] [2][3] [4] the only real solution is to get the software from the projects own website. This works well for the big projects that have their own sites such as Firefox, Gimp, VLC just to name a few.

This isn’t the only possible solution, although it is probably the easiest one. In the last few years computers have not gotten that much more powerful but their cost and the cost of disks and ram have come even further down in price. My next article will be about using software to create a virtual computers for testing software or risky web sites.

References

[0] http://www.computerworld.com/article/2890512/bloatware-free-windows-computers.html

[1] http://www.howtogeek.com/198622/heres-what-happens-when-you-install-the-top-10-download.com-apps/

[2] http://www.hanselman.com/blog/DownloadWrappersAndUnwantedSoftwareArePureEvil.aspx

[3] https://dottech.org/23420/cnet-crapware/

[4] http://www.howtogeek.com/207692/yes-every-freeware-download-site-is-serving-crapware-heres-the-proof/

Posted in Soapbox | Tagged paranoid, virus | Comments Off

just making it – a 4x4x4 led cube

Posted on September 24, 2016 by DocDock

Some time back, I had a fair amount lot of free time and did watch things on you tube. A friend of mine showed a number of cool LED cubes that were trending at that time.

It seemed so cool and considering our combined backgrounds a natural that we should try and create our own control board for our own LED cube. He was the hardware engineer so I left him to that task while I thought about patterns and programming the cube.

Yet, despite not being the hardware type I thought I could create my own, albeit less sophisticated cube, with only a small bit of effort.

Come on just how hard could it really be?

Bring in the pie’s

Part of the reason I thought this would be so simple is that I was also playing around with the “Raspberry Pi”. The specifications for the pi were not all that impressive when compared to my laptop but it was a truly amazingly affordable and small computer that also offered a comfortable Linux distribution for running it.

The Raspberry Pi that I was using is basically just a small computer with the power of a smart phone from 6 years ago.

ARM11 700 mhz
SoC is a Broadcom BCM2835
512MB RAM
2 USB ports
Ethernet port
3.5mm jack for audio out
HDMI

The operating system is a derivative of Debian. Communicating with the Raspberry Pi can be done just like any other computer over the Ethernet or perhaps by using a WiFi dongle.

Communication between the Raspberry Pi and other external peripherals is done via the general purpose I/O pins. The GPIO is the break out of the functionality in the Broadcom BCM2835 system on a chip which in additional to other functionality also includes the support for I2C and SPI as well as UHF transmission.

I2C and SPI are two different but common communication bus protocols for communication between micro-controllers and other components or peripherals. The protocol allows a component to send commands to a receiving component. This is a really convenient way to delegate tasks to other devices, but inter-device communication is not the only advantage of the GPIO.

The other advantage of the GPIO pins is that it is possible to use it to turn on or off something connected to one of these pins. This can be used to enable or disable other devices or even simple things like turning on a motor or a LED.

How LED cubes work

LED cubes create neat images by turning on LED’s in rapid succession in order to creates letters or patterns. The power required to turn on all the lights becomes more and more considerable as the cubes get larger and larger. A 3x3x3 cube can easily be powered by a 9 volt battery but a 8x8x8 or 16x16x16 cube would require a an actual power supply.

The thing that is the same no matter of the size is that the cubes only turn on a single led at a time and rely on persistence of vision to fool the eye into believing that all the LED’s in the pattern are on at the same time.

Persistence of vision principle is essentially the same concept underpinning “moving pictures” aka cartoons. The eye cannot distinguish between changes that occur too quickly and so it will see it as a single picture. Thus when the different LED’s in a cube switch rapidly on and off the eye simply sees the LED’s were on.

My own cubes logic

The first thing that I needed to be consider was how to power each led. Four levels of 16 LED’s gives a total of 64 LED’s or 128 legs (anode and cathode) for the cube. The total number of GPIO pins available on my Raspberry Pi is 26. Even if I could use every single one of these pins this would be nowhere near enough pins for this small project.

This is a very important point as not every one of these pins is even available to be used. Some of these pins are used for I2C or power or ground which leaves me severely lacking if the goal is that the Raspberry Pi is to drive everything directly.

Yet, I cannot be the only person or manufacturer to have encountered this problem and indeed I was not.

23017 IO Extender

The 23017 IO extender has 16 data legs which can be turned on or off. One of these chips alone won’t be enough to deal with 128 legs but there are some clever simplifications to the overall design that will allows me to use two of these chips to drive the rest of the cube.

Because we only need to turn on one led at a time, I can use the IO extender for each of the LED columns. Thus if I provide power for each level, as needed, and allow the current to flow through the led controlled by each column then I can turn on or off any led at any spot on the cube.

This means that I will need to use one of the 23017’s only to control the columns but then I need a way to control which levels get the power. This is done by using a second 23017 to control which level will receive the power.

How this works is as follows. A light bulb or a LED cannot light unless it receives a path to power and ground. When this happens the electricity flows through lighting up the element. Controlling a light bulb is done using a normal wall switch, and this can be done for a LED as well. However, for a LED in a circuit another possibility is to prevent the power from flowing.

This is done if either no current is provided for the input (anode) of the LED or no ground (cathode) provided to complete the circuit. If both of these pins are set to ground or both are provided voltage the current will not flow and the LED will not light up.

This how my cube controls the individual LED’s. Once the program decides which LED to light up, it will provide current to that level and will set the cathode for that LED to ground.

Well, this is how my circuit behaves, but the Raspberry Pi is not toggling the state of the LED’s directly but actually using I2C to send a command to both of my 23017’s to enable each LED. Each 23017 is given a different address so it can be controlled directly. This is done by setting the pins 15 – 17 to a unique value for the 23017 chips that you are using.

MCP 23017

Pin	Name	Description
1	GPB0	I/O port B
2	GPB1	I/O port B
3	GPB2	I/O port B
4	GPB3	I/O port B
5	GPB4	I/O port B
6	GPB5	I/O port B
7	GPB6	I/O port B
8	GPB7	I/O port B
9	VDD	5 volt power
10	VSS	ground
11	NC	unused
12	SCL	Clock pin
13	SDA	Data pin
14	NC	unused
15	A0	Address
16	A1	Address
17	A2	Address
18	Reset	Reset
19	INTB	Interrupt B
20	INTA	Interrupt A
21	GPA0	I/O port A
22	GPA1	I/O port A
23	GPA2	I/O port A
24	GPA3	I/O port A
25	GPA4	I/O port A
26	GPA5	I/O port A
27	GPA6	I/O port A
28	GPA7	I/O port A

This table is intended to give a brief overview. To have a full understanding you will need to reference the specification sheet for this chip.

Reading through the I2C specification of how exactly the bus functions is fascinating but perhaps a small it intimidating at the same time. The nice thing for the application developer is that, more likely than not, this is a low level function that you call with some parameters.

In my case I was able to find a open source library on the internet which supported the BMC2835 chip which included an I2C support.

Circuit diagram

Despite how complex this circuit may appear it is simply two IO extender chips, three capacitors and twenty resisters.

Somewhat contrary to what you might think, the circuit diagram wasn’t created so we could create the following board.

ledcube-board

We created the board layout in order to design an optimal layout for the individual connections.

There is really very little needs to be described for this circuit. On the left side of the circuit is a simple five line connector connector. These lines will connect to the Raspberry Pi which will provide power, ground and the SCL and SDA in order to use I2C to communicate with the 23017 chips.

Building the cube

Building the actual LED cube is both the easiest and the hardest part of this project.

layers

On one hand it should not be overly difficult to solder together the LED’s. It is just a matter of soldering each LED to a wire. This should be made even easier by creating a template to hold all of the LED’s still in the proper relative positions to each other.

Indeed this template makes it relatively easy to create a plane of LED’s. You can take as much time as you need and there is no need of any additional tools or assistants to make these planes. One minor inconvenience is to find your own way to create straight wires.

If the wires are not straight then the rows or columns won’t be either. While preparing for this I have hit upon three different manners to prepare the wires.

pull the wires
use a drill
use a couple of boards

The first two methods pretty much require a vice. Simply connect the wire to the vice and either pull the wire ever so slightly or put the other end into a drill and briefly run the drill. Both of these methods actually stretch the wire slightly which is what causes it to straighten out.

I only saw a single video on the topic of using boards to straighten wires, and somehow I lost that link. The idea is pretty simple, you need a flat board and a small thin board that is at least as long as the wire and approximately six cm wide.

dav

Just place the wire on the flat board and set the smaller board on top of it. Rub the wire left and right as many times as it takes for it to straighten out. This is a very convenient method as long as the wires don’t need to be too long.

Assemble the layers

The most challenging part is the assembly of the different layers. I don’t think using the word trivial is correct for creating the layers but you can work and re-work them until you are happy. The hard task is the assembly.

The reason this is so difficult is that this method of assembly requires that you use your soldering iron for each layer and you need to solder in the middle. This is fairly easy for the first one but gets more complex as more layers are added.

In addition, you have to be extremely careful not to accidentally touch any of the wires with your soldering iron otherwise too much heat may be transmitted to the nearest solder joint. If the LED’s or wires have any pressure pushing them apart this heat may be all it takes to cause a previously soldered part to either come loose or become a weak joint / loose connection. It is very difficult to re-solder these connections once the cube is built. This would become even more problematic the larger the cube is.

The level of difficulty assembling in this manner due in part to how densely packed the cube should be. This is both the horizontal and the vertical density. Two difficulties present themselves.

The first difficulty is how much spacing to keep between each plane. The second and related problem is ensuring that the spacing is kept uniformly for each layer and for all subsequent layers.

I didn’t do it with my cube, I used wires for most of the structure, but quite a few other cubes are built using the actual cathodes and anodes to build up the cube structure. If this were done it would help considerably with the spacing problems.

dav

Drawbacks of this design

My father always said that you should use the correct tool for the job. In this case, I imagine that the right tool would have been an electrical engineer.

As long as you already have a Raspberry Pi, my design is a low cost, low component count that can easily be put together either with a breadboard or with a small amount of soldering. Yet, the Raspberry Pi is not a power plant, there are limitations to how much power it can produce.

The power required to turn on all the lights becomes more and more considerable as the cubes get larger . A 3x3x3 cube can easily be powered by a 9 volt battery but a 8x8x8 or 16x16x16 cube would require a an actual power supply.

My solution is using I2C for communication with the LED’s via the 23017. The only problem is that I found this communication to be pretty slow. My LED’s were not as bright as a lot of the other LED cubes that you can see on youtube.

It was because of the power limitations on how much current can be channeled through the MPC23017 that caused me in the end to take a conservative view and cause me to pick the low power LED’s. The good news is that if I turn on every LED in my cube at the same time it is pretty much within the limitations of the power that can be driven by the 23017. The cube looks good in a dark room but you cannot hardly see it in normal lighting conditions.

I was also a bit disappointed in exactly how symmetrical my cube was. Not bad for a first attempt but I was hoping for better. Unfortunately, the actual LED cube is the product of its creator and the result is unrelated to using a kit or creating everything from scratch. The best results are most likely to be attributed to slow and steady work and a great deal of thought and preparation.

In retrospect the biggest flaw in my design was originally its biggest strength. The nice thing about using the Raspberry Pi was I simply used secure shell to connect to the computer and wrote, compiled, and debugged the code in one easy step.

The only problem is that the Raspberry Pi is a computer not some off the shelf device with a micro-controller. This means that when you turn it on, it takes a while before the computer is booted up and the application can start.

The startup isn’t the problem, the problem when it comes down to shutting down the cube. The Raspberry Pi doesn’t doesn’t have an on / off button so to shut it off I need to open up a secure shell to the Raspberry Pi and then issue a shutdown. Once the device is shutdown then unplug the power supply.

Because the Raspberry Pi isn’t really doing anything special it could have been replaced with either an Arduino or a smaller Arduino and my custom controller board.

Alternatives

The internet is full of instructional videos, instructions and photos of how to make your own LED cube.

One of the difficulties that I mentioned was that building a cube from horizontal layers. The next cube that I would make would take a one of two different alternatives. The first is to simply solder together an entire column of LED’s and then start to solder them to the board and to each other. The second alternative is somewhat similar. Simply solder together vertical panels and solder them together.

Show code (2,519 More Words)

Code

picube.c

#include <bcm2835.h>

cat picube.c
#include <bcm2835.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <signal.h>
#include "picube.h"

uint cathode = 0x20;
uint anode = 0x21;
uint IOCON = 0x0A;
uint IODIRA = 0x00;
uint IODIRB = 0x01;
uint OLATA = 0x14;

int mapping[4][4];
int zmapping[4];

int debug = 0;
int verbose = 0;

void sigint_handler()
{
 printf("unexpected request, shutting down program\n");
 final_bcm2835();
}

void init_mapping()
{
 mapping[0][1] = 3;
 mapping[0][0] = 7;
 mapping[0][3] = 11;
 mapping[0][2] = 15;

 mapping[1][1] = 2;
 mapping[1][0] = 6;
 mapping[1][3] = 10;
 mapping[1][2] = 14;

 mapping[2][1] = 1;
 mapping[2][0] = 5;
 mapping[2][3] = 9;
 mapping[2][2] = 13;

 mapping[3][1] = 0;
 mapping[3][0] = 4;
 mapping[3][3] = 8;
 mapping[3][2] = 12;

 if (debug != 0)
 {
 int x,y;
 for (x = 0; x < 4; x++)
 {
 for (y = 0; y < 4; y++)
 printf("%02d ",mapping[x][y]);
 printf("\n");
 }
 }

 zmapping[0] = ZDIM0;
 zmapping[1] = ZDIM1;
 zmapping[2] = ZDIM2;
 zmapping[3] = ZDIM3;
}

/*
** a bit of checking of return codes
*/
void check_retcode(int status)
{
 switch (status)
 {
 case BCM2835_I2C_REASON_OK:
 //printf("Code: Ok\n");
 break;
 case BCM2835_I2C_REASON_ERROR_NACK:
 printf("Code: Received a NACK\n");
 break;
 case BCM2835_I2C_REASON_ERROR_CLKT:
 printf("Code: Received Clock Stretch Timeout\n");
 break;
 case BCM2835_I2C_REASON_ERROR_DATA:
 printf("Code: Not all data is sent / received\n");
 break;
 default:
 printf("Code: unknown\n");
 break;
 }
}

// more for debugging than anything else.
void turn_on_entire_cube()
{
 if (verbose != 0)
 printf("turning on whole cube\n");

 // turn on all cathodes
 bcm2835_i2c_setSlaveAddress(cathode);
 char cmd7[] = { OLATA, 0x00, 0x00};
 check_retcode(bcm2835_i2c_write(cmd7,sizeof(cmd7)));

 // turn on all layers (anodes)
 char cmd9[] = { OLATA, 0xF0 };
 bcm2835_i2c_setSlaveAddress(anode);
 check_retcode(bcm2835_i2c_write(cmd9,sizeof(cmd9)));
 sleep(1);
}

void turn_off_entire_cube()
{
 if (debug != 0)
 printf("turning off whole cube\n");

 // turn off all layers
 char cmd10[] = { OLATA, 0x00 };
 bcm2835_i2c_setSlaveAddress(anode);
 check_retcode(bcm2835_i2c_write(cmd10,sizeof(cmd10)));
}

void init_bcm2835()
{
 if (!bcm2835_init())
 {
 printf("failed on init\n");
 exit(1);
 }

 bcm2835_i2c_begin();

 bcm2835_i2c_set_baudrate(400000);
}

void init_cathode()
{
 // point library to cathode
 bcm2835_i2c_setSlaveAddress(cathode);

 // set bit 7 = 0 use consecutive mapping
 // set bit 5 = 1 (address increment) to on
 // all others dont care.
 char cmd[] = { IOCON, 0x20 };
 check_retcode(bcm2835_i2c_write(cmd,sizeof(cmd)));

 // all pins output
 char cmd2[] = { IODIRA, 0x00, 0x00 };
 check_retcode(bcm2835_i2c_write(cmd2,sizeof(cmd2)));

#if 0
 // all pins output
 char cmd3[] = { IODIRB, 0x00 };
 check_retcode(bcm2835_i2c_write(cmd3,sizeof(cmd3)));
#endif
}

void init_anode()
{
 // point library to anode
 bcm2835_i2c_setSlaveAddress(anode);

 char cmd6[] = { IOCON, 0x20 };
 check_retcode(bcm2835_i2c_write(cmd6,sizeof(cmd6)));

 // all pins output
 char cmd4[] = { IODIRA, 0x00, 0x00 };
 check_retcode(bcm2835_i2c_write(cmd4,sizeof(cmd4)));

#if 0
 // all pins output
 char cmd5[] = { IODIRB, 0x00 };
 check_retcode(bcm2835_i2c_write(cmd5,sizeof(cmd5)));
#endif
}

void init_microprocessor()
{
 init_cathode();
 init_anode();
}

void light_mask_z(int mask, int zmask)
{
 int bitmask, invertmask, lowerx, upperx;

 /*
 ** first deal with setting up the cathodes
 */

 if (debug != 0 )
 printf("mask %04x z%2d\n",mask,zmask);

 // which bits should be on
 bitmask = mask;

 // then we invert bits
 invertmask = ~bitmask & 0xffff;

 // break it up for microprocessor
 lowerx = invertmask & 0x00ff;
 upperx = (invertmask & 0xff00) >> 8;

 bcm2835_i2c_setSlaveAddress(cathode);

 char cmd7[] = { OLATA, upperx, lowerx};
 check_retcode(bcm2835_i2c_write(cmd7,sizeof(cmd7)));

 /*
 ** now deal with setting up the anodes
 */
 bcm2835_i2c_setSlaveAddress(anode);

 // setup our layer
 // set which layers should be on

 bitmask = zmask;
 char cmd8[] = { OLATA, bitmask & 0xff };
 check_retcode(bcm2835_i2c_write(cmd8,sizeof(cmd8)));
}

// x is 0 - 3
// y is 0 - 3
// z is 0 - 3
void light_x_y_z(int X, int Y, int Z)
{
 int bitmask, invertmask, lowerx, upperx;
 int mappedpin;

 /*
 ** first deal with setting up the cathodes
 */
 mappedpin = mapping[X][Y];

 if (debug != 0)
 printf("x%2d(%d) y%2d z%2d\n",X,mappedpin,Y,Z);

 // first set the bit
 bitmask = ( 1 << mappedpin );

 // then we invert bits
 invertmask = ~bitmask & 0xffff;

 // break it up for microprocessor
 lowerx = invertmask & 0x00ff;
 upperx = (invertmask & 0xff00) >> 8;

// printf("%2d %04x %04x %02x %02x\n",mappedpin,bitmask,invertmask,upperx,lowerx);

 bcm2835_i2c_setSlaveAddress(cathode);

 char cmd7[] = { OLATA, upperx, lowerx};
 check_retcode(bcm2835_i2c_write(cmd7,sizeof(cmd7)));

 /*
 ** now deal with setting up the anodes
 */
 bcm2835_i2c_setSlaveAddress(anode);

 // setup our layer
 // set which layers should be on

 bitmask = ( 0x10 << Z );
 char cmd8[] = { OLATA, bitmask & 0xff };
 check_retcode(bcm2835_i2c_write(cmd8,sizeof(cmd8)));
}

void rotate_layers()
{
 if (verbose != 0)
 printf("rotate layers\n");

 // enable
 int idx;
 bcm2835_i2c_setSlaveAddress(anode);
 for (idx = 0; idx < 4; idx++)
 {
 // actually write something out, will turn on
 // one layer for any leds that are setup

 char cmd8[] = { OLATA, 0x10 << idx };

 if (debug != 0)
 printf("cmd8 %x %x\n",idx, 0x10 << idx);
 check_retcode(bcm2835_i2c_write(cmd8,sizeof(cmd8)));
 sleep(2);
 }
}

void circle_chase_base(int speed, double divisor)
{
 int x,y;
 int z = 1;
 int iterations;

 for (iterations = 0; iterations < 10; iterations++)
 {
//printf("speed %d\n",speed);
 for (z = 0; z < 4; z++)
 {
 for (x = 0; x < 4; x++)
 {
 light_x_y_z(x,0,z);
 delay(speed);
 }
 for (y = 1; y < 4; y++)
 {
 light_x_y_z(3,y,z);
 delay(speed);
 }

 for (x = 2; x > 0;x--)
 {
 light_x_y_z(x,3,z);
 delay(speed);
 }
 for (y = 3; y > 0; y--)
 {
 light_x_y_z(0,y,z);
 delay(speed);
 }
 }
 speed = (int)(divisor * (double)speed);
 }
}

void fast_circle_chase()
{
 if (verbose != 0)
 printf("fast circle chase\n");

 circle_chase_base(15,1);
}

void circle_chase()
{
 if (verbose != 0)
 printf("normal circle chase\n");

 circle_chase_base(200,0.65);
}

void one_after_another()
{
 int count,x,y,z;

 if (verbose != 0)
 printf("one led after another\n");

 for (count = 0; count < 1; count++)
 {
 for (z = 0; z < 4; z++)
 for (y = 0; y < 4; y++)
 for (x = 0; x < 4; x++)
 {
 light_x_y_z(x,y,z);
 delay(30);
 }
 }
}

void top_bottom_rotate_side()
{
 int counter;

 if (verbose != 0)
 printf("top bottom rotate\n");

 for (counter = 0; counter < 5; counter++)
 {
 light_mask_z(YCOL0,ZDIM_TOPBOT); delay(250);
 light_mask_z(XROW3,ZDIM_TOPBOT); delay(250);
 light_mask_z(YCOL3,ZDIM_TOPBOT); delay(250);
 light_mask_z(XROW0,ZDIM_TOPBOT); delay(250);
 }
}

void side_corkscrew()
{
 int counter;
 if (verbose != 0)
 printf("side corkscrew\n");

 for (counter = 0; counter < 5; counter++)
 {
 // sides
 light_mask_z(YCOL0,ZDIM0); delay(250);
 light_mask_z(XROW3,ZDIM1); delay(250);
 light_mask_z(YCOL3,ZDIM2); delay(250);
 light_mask_z(XROW0,ZDIM3); delay(250);

 // column CA7
 light_mask_z(CA07,ZDIM_ALL); delay(250);
 }
}

void rotate_takeoff()
{
 if (verbose != 0)
 printf("rotate take off\n");

 light_x_y_z(3, 0, 0); // A
 light_x_y_z(2, 1, 0);
 light_x_y_z(1, 2, 0);
 light_x_y_z(0, 3, 0);
 sleep (200);

 light_x_y_z(2, 0, 0); // B
 light_x_y_z(2, 1, 0);
 light_x_y_z(2, 2, 0);
 light_x_y_z(2, 3, 0);
 sleep (200);

 light_x_y_z(1, 0, 0); // C
 light_x_y_z(1, 1, 0);
 light_x_y_z(1, 2, 0);
 light_x_y_z(1, 3, 0);
 sleep (200);

 light_x_y_z(0, 1, 0); // D
 light_x_y_z(1, 1, 0);
 light_x_y_z(2, 1, 0);
 light_x_y_z(3, 1, 0);
 sleep (200);

 light_x_y_z(0, 2, 0); // E
 light_x_y_z(1, 2, 0);
 light_x_y_z(2, 2, 0);
 light_x_y_z(3, 2, 0);
 sleep (200);

 light_x_y_z(0, 0, 0); // F
 light_x_y_z(1, 1, 0);
 light_x_y_z(2, 2, 0);
 light_x_y_z(3, 3, 0);
 sleep (200);

 light_x_y_z(1, 1, 0); // G
 light_x_y_z(2, 1, 0);
 light_x_y_z(1, 2, 0);
 light_x_y_z(2, 2, 0);
 sleep (200);

 light_x_y_z(1, 1, 0); // G
 light_x_y_z(2, 1, 0);
 light_x_y_z(1, 2, 0);
 light_x_y_z(2, 2, 0);
 sleep (200);

 // H
 light_mask_z(OUTERRING,0); delay(250);

 // I
 light_mask_z(CORNERS,0); delay(250);
}

void fireworks()
{
 int x, y;

 if (verbose != 0)
 printf("fireworks\n");

 for (x = 1; x <= 2; x++)
 for (y = 1; y <= 2; y++)
 {
 light_x_y_z(x, y, 0);
 delay(100);

 light_x_y_z(x, y, 0);
 light_x_y_z(x, y, 1);
 delay(100);

 light_x_y_z(x, y, 0);
 light_x_y_z(x, y, 1);
 light_x_y_z(x, y, 2);
 delay(100);

 light_x_y_z(x, y, 0);
 light_x_y_z(x, y, 1);
 light_x_y_z(x, y, 2);
 light_x_y_z(x, y, 3);
 delay(100);

 light_x_y_z(x, y, 3);
 delay(25);

 light_mask_z(INNERRING,ZDIM3);
 delay(150);
 light_mask_z(OUTERRING,ZDIM3);
 delay(150);

 // falling pieces
 int zdim ;
 for (zdim = 3; zdim >= 0; zdim--)
 {
 if (zdim % 2 == 1)
 light_mask_z( CA05 | CA12 | CA09 ,zmapping[zdim]);
 else
 light_mask_z(CA07 | CA04 | CA08 | CA11 ,zmapping[zdim]);
 delay(75 + zdim * 25);
 }
 turn_off_entire_cube();
 }
}

void helicopter()
{
 int smallwait[] = {110, 90, 65, 40 };
 int zdim,idx;

 if (verbose != 0)
 printf("helicopter\n");

 for (zdim = 0; zdim < 4; zdim++)
 {
 for (idx = 0; idx <= zdim+4; idx++)
 {
 light_mask_z(PATTERN_A,zmapping[zdim]); delay(smallwait[zdim]);
 light_mask_z(PATTERN_B,zmapping[zdim]); delay(smallwait[zdim]);
 light_mask_z(PATTERN_C,zmapping[zdim]); delay(smallwait[zdim]);
 light_mask_z(PATTERN_F,zmapping[zdim]); delay(smallwait[zdim]);
 light_mask_z(PATTERN_D,zmapping[zdim]); delay(smallwait[zdim]);
 light_mask_z(PATTERN_E,zmapping[zdim]); delay(smallwait[zdim]);
 }
 }

 for (zdim = 0; zdim < 4; zdim++)
 {
 light_mask_z(PATTERN_G,zmapping[zdim]); delay(1000/(zdim+1));
 light_mask_z(PATTERN_H,zmapping[zdim]); delay(750/(zdim+1));
 }

 for (zdim = 3; zdim >= 0; zdim--)
 {
 light_mask_z(PATTERN_I,zmapping[zdim]); delay(300);
 }
}

void x_rotating_plate()
{
 int idx,rep,smallwait = 10;
 if (verbose != 0)
 printf("x rotating plate\n");

 for (idx = 0; idx < 10; idx++)
 {
 for (rep = 0; rep < 5; rep++)
 {
 light_mask_z(XROW3,ZDIM3); delay(smallwait);
 light_mask_z(XROW2,ZDIM2); delay(smallwait);
 light_mask_z(XROW1,ZDIM1); delay(smallwait);
 light_mask_z(XROW0,ZDIM0); delay(smallwait);
 }
 for (rep = 0; rep < 5; rep++)
 {
 light_mask_z(XROW3,ZDIM1); delay(smallwait);
 light_mask_z(XROW2,ZDIM1); delay(smallwait);
 light_mask_z(XROW1,ZDIM1); delay(smallwait);
 light_mask_z(XROW0,ZDIM1); delay(smallwait);
 }
 for (rep = 0; rep < 5; rep++)
 {
 light_mask_z(XROW3,ZDIM2); delay(smallwait);
 light_mask_z(XROW2,ZDIM2); delay(smallwait);
 light_mask_z(XROW1,ZDIM2); delay(smallwait);
 light_mask_z(XROW0,ZDIM2); delay(smallwait);
 }
 for (rep = 0; rep < 5; rep++)
 {
 light_mask_z(XROW0,ZDIM3); delay(smallwait);
 light_mask_z(XROW1,ZDIM2); delay(smallwait);
 light_mask_z(XROW2,ZDIM1); delay(smallwait);
 light_mask_z(XROW3,ZDIM0); delay(smallwait);
 }
 for (rep = 0; rep < 5; rep++)
 {
 light_mask_z(XROW1,ZDIM3); delay(smallwait);
 light_mask_z(XROW1,ZDIM2); delay(smallwait);
 light_mask_z(XROW1,ZDIM1); delay(smallwait);
 light_mask_z(XROW1,ZDIM0); delay(smallwait);
 }
 for (rep = 0; rep < 5; rep++)
 {
 light_mask_z(XROW2,ZDIM3); delay(smallwait);
 light_mask_z(XROW2,ZDIM2); delay(smallwait);
 light_mask_z(XROW2,ZDIM1); delay(smallwait);
 light_mask_z(XROW2,ZDIM0); delay(smallwait);
 }
 turn_off_entire_cube();
 }
}

void y_rotating_plate()
{
 int idx,rep,smallwait = 10;
 if (verbose != 0)
 printf("y rotating plate\n");

 for (idx = 0; idx < 10; idx++)
 {

 for (rep = 0; rep < 5; rep++)
 {
 light_mask_z(YCOL3,ZDIM3); delay(smallwait);
 light_mask_z(YCOL2,ZDIM2); delay(smallwait);
 light_mask_z(YCOL1,ZDIM1); delay(smallwait);
 light_mask_z(YCOL0,ZDIM0); delay(smallwait);

 }
 for (rep = 0; rep < 5; rep++)
 {
 light_mask_z(YCOL3,ZDIM1); delay(smallwait);
 light_mask_z(YCOL2,ZDIM1); delay(smallwait);
 light_mask_z(YCOL1,ZDIM1); delay(smallwait);
 light_mask_z(YCOL0,ZDIM1); delay(smallwait);
 }
 for (rep = 0; rep < 5; rep++)
 {
 light_mask_z(YCOL3,ZDIM2); delay(smallwait);
 light_mask_z(YCOL2,ZDIM2); delay(smallwait);
 light_mask_z(YCOL1,ZDIM2); delay(smallwait);
 light_mask_z(YCOL0,ZDIM2); delay(smallwait);
 }
 for (rep = 0; rep < 5; rep++)
 {
 light_mask_z(YCOL0,ZDIM3); delay(smallwait);
 light_mask_z(YCOL1,ZDIM2); delay(smallwait);
 light_mask_z(YCOL2,ZDIM1); delay(smallwait);
 light_mask_z(YCOL3,ZDIM0); delay(smallwait);
 }
 for (rep = 0; rep < 5; rep++)
 {
 light_mask_z(YCOL1,ZDIM3); delay(smallwait);
 light_mask_z(YCOL1,ZDIM2); delay(smallwait);
 light_mask_z(YCOL1,ZDIM1); delay(smallwait);
 light_mask_z(YCOL1,ZDIM0); delay(smallwait);
 }
 for (rep = 0; rep < 5; rep++)
 {
 light_mask_z(YCOL2,ZDIM3); delay(smallwait);
 light_mask_z(YCOL2,ZDIM2); delay(smallwait);
 light_mask_z(YCOL2,ZDIM1); delay(smallwait);
 light_mask_z(YCOL2,ZDIM0); delay(smallwait);
 }
 turn_off_entire_cube();
 }
}

// rotates around the z axis
void z_rotating_plate()
{
 int smallwait[] = {250, 200, 150, 100 };
 int zdim,idx;
 if (verbose != 0)
 printf("z rotating plate\n");

 for (idx = 0; idx < 4; idx++)
 for (zdim = 0; zdim < 4; zdim++)
 {
 for (idx = 0; idx <= zdim; idx++)
 {
 light_mask_z(PATTERN_A,ZDIM_ALL); delay(smallwait[zdim]);
 light_mask_z(PATTERN_B,ZDIM_ALL); delay(smallwait[zdim]);
 light_mask_z(PATTERN_C,ZDIM_ALL); delay(smallwait[zdim]);
 light_mask_z(PATTERN_F,ZDIM_ALL); delay(smallwait[zdim]);
 light_mask_z(PATTERN_D,ZDIM_ALL); delay(smallwait[zdim]);
 light_mask_z(PATTERN_E,ZDIM_ALL); delay(smallwait[zdim]);
 }
 }
}

void wire_frame()
{
 int iter;

 if (verbose != 0)
 printf("wire frame\n");

 for (iter = 0; iter < 200; iter++)
 {
 light_mask_z(OUTERRING,ZDIM0); delay(10);
 light_mask_z(CORNERS,ZDIM1); delay(10);
 light_mask_z(CORNERS,ZDIM2); delay(10);
 light_mask_z(OUTERRING,ZDIM3); delay(10);
 }
}

void shrinking_cube()
{
 int iter,idx;
 int redraw = 20;

 if (verbose != 0)
 printf("shrinking cube\n");

 for (idx = 0; idx < 5; idx++)
 {
 // bigest cube
 for (iter = 0; iter < redraw; iter++)
 {
 light_mask_z(XROW0 | XROW3 | YCOL0 | YCOL3, ZDIM0); delay(10);
 light_mask_z(CA07 | CA11 | CA08 | CA04,ZDIM1); delay(10);
 light_mask_z(CA07 | CA11 | CA08 | CA04,ZDIM2); delay(10);
 light_mask_z(XROW0 | XROW3 | YCOL0 | YCOL3, ZDIM3); delay(10);
 }

 // next bigest cube
 for (iter = 0; iter < redraw; iter++)
 {
 light_mask_z(CA05 | CA06 | CA07 | CA03 | CA01 | CA15 | CA14 | CA13, ZDIM0);
 delay(10);

 light_mask_z(CA03 | CA06 | CA01 | CA14, ZDIM1);
 delay(10);

 light_mask_z(CA05 | CA06 | CA07 | CA03 | CA01 | CA15 | CA14 | CA13, ZDIM2);
 delay(10);
 }

 // next biggest
 for (iter = 0; iter < redraw; iter++)
 {
 light_mask_z(CA07 | CA06 | CA03 | CA02, ZDIM0); delay(10);
 light_mask_z(CA07 | CA06 | CA03 | CA02, ZDIM1); delay(10);
 }

 // smallest cube, actually just a point
 for (iter = 0; iter < redraw; iter++)
 {
 light_mask_z(CA07, ZDIM0); delay(10);
 }

 delay(100);

 // next biggest
 for (iter = 0; iter < redraw; iter++)
 {
 light_mask_z(CA07 | CA06 | CA03 | CA02, ZDIM0); delay(10);
 light_mask_z(CA07 | CA06 | CA03 | CA02, ZDIM1); delay(10);
 }

 // next bigest cube
 for (iter = 0; iter < redraw; iter++)
 {
 light_mask_z(CA05 | CA06 | CA07 | CA03 | CA01 | CA15 | CA14 | CA13, ZDIM0);
 delay(10);

 light_mask_z(CA03 | CA06 | CA01 | CA14, ZDIM1);
 delay(10);

 light_mask_z(CA05 | CA06 | CA07 | CA03 | CA01 | CA15 | CA14 | CA13, ZDIM2);
 delay(10);
 }

 // bigest cube
 for (iter = 0; iter < redraw; iter++)
 {
 light_mask_z(XROW0 | XROW3 | YCOL0 | YCOL3, ZDIM0); delay(10);
 light_mask_z(CA07 | CA11 | CA08 | CA04,ZDIM1); delay(10);
 light_mask_z(CA07 | CA11 | CA08 | CA04,ZDIM2); delay(10);
 light_mask_z(XROW0 | XROW3 | YCOL0 | YCOL3, ZDIM3); delay(10);
 }
 }
}

void floating_cube(int pause)
{
 int iter;
 int count = 2;
 int cubes[] = { CA04 | CA00 | CA05 | CA01, // 0
 CA05 | CA01 | CA06 | CA02, // 1
 CA06 | CA02 | CA07 | CA03, // 2
 CA02 | CA14 | CA03 | CA15, // 3
 CA14 | CA10 | CA15 | CA11, // 4
 CA13 | CA09 | CA14 | CA10, // 5
 CA12 | CA08 | CA13 | CA09, // 6
 CA00 | CA12 | CA01 | CA13, // 7
 CA01 | CA13 | CA02 | CA14, // 8
 };
 int lvls[] = { ZDIM0 | ZDIM1, ZDIM1 | ZDIM2, ZDIM2 | ZDIM3 };

 // next biggest
 for (iter = 0; iter < count; iter++)
 {
 light_mask_z(cubes[0],lvls[0]); delay(pause);
 light_mask_z(cubes[1],lvls[0]); delay(pause);
 light_mask_z(cubes[1],lvls[1]); delay(pause);
 light_mask_z(cubes[2],lvls[1]); delay(pause);
 light_mask_z(cubes[8],lvls[1]); delay(pause);
 light_mask_z(cubes[8],lvls[2]); delay(pause);
 light_mask_z(cubes[3],lvls[2]); delay(pause);

 light_mask_z(cubes[4],lvls[2]); delay(pause);
 light_mask_z(cubes[5],lvls[1]); delay(pause);
 light_mask_z(cubes[8],lvls[1]); delay(pause);

 light_mask_z(cubes[6],lvls[0]); delay(pause);
 light_mask_z(cubes[7],lvls[1]); delay(pause);
 light_mask_z(cubes[8],lvls[0]); delay(pause);

 light_mask_z(cubes[0],lvls[0]); delay(pause);
 light_mask_z(cubes[8],lvls[1]); delay(pause);
 light_mask_z(cubes[4],lvls[2]); delay(pause);
 light_mask_z(cubes[3],lvls[2]); delay(pause);
 light_mask_z(cubes[2],lvls[1]); delay(pause);
 light_mask_z(cubes[1],lvls[1]); delay(pause);
 light_mask_z(cubes[0],lvls[0]); delay(pause);

 }
}

void final_bcm2835()
{
 // make sure we turn off the leds
 turn_off_entire_cube();

 // shutdown in an orderly manner
 bcm2835_i2c_end();
 bcm2835_close();
}

int main(int argc, char **argv)
{
 signal(SIGINT,sigint_handler);
 init_mapping();

 init_bcm2835();

 init_microprocessor();

 int loopcounter ;
 for (loopcounter = 0; loopcounter < 10; loopcounter++)
 {
//#if 0

loopcounter = 1;
 turn_on_entire_cube();
 sleep(2);

 one_after_another();
 sleep(1);

 floating_cube(700);
 sleep(1);

 circle_chase();
 sleep(1);

 side_corkscrew();
 sleep(1);

 wire_frame();

 top_bottom_rotate_side();
 sleep(1);

 fast_circle_chase();
 sleep(1);

 helicopter();
 sleep(1);

 fireworks();
 sleep(1);

 x_rotating_plate();
 floating_cube(250);

 y_rotating_plate();
 shrinking_cube();

 z_rotating_plate();

//printf("%d\n",loopcounter);
//#endif
 shrinking_cube();
 }

 final_bcm2835();
 return 0;
}

picube.h

#ifndef _PICUBE_H_
#define _PICUBE_H_

extern void final_bcm2835();

#define CA07 0x0080
#define CA06 0x0040
#define CA05 0x0020
#define CA04 0x0010
#define CA03 0x0008
#define CA02 0x0004
#define CA01 0x0002
#define CA00 0x0001

#define CA15 0x8000
#define CA14 0x4000
#define CA13 0x2000
#define CA12 0x1000
#define CA11 0x0800
#define CA10 0x0400
#define CA09 0x0200
#define CA08 0x0100

#define YCOL0 CA07 | CA03 | CA15 | CA11
#define YCOL1 CA06 | CA02 | CA14 | CA10
#define YCOL2 CA05 | CA01 | CA13 | CA09
#define YCOL3 CA04 | CA00 | CA12 | CA08

#define XROW0 CA07 | CA06 | CA05 | CA04
#define XROW1 CA03 | CA02 | CA01 | CA00
#define XROW2 CA15 | CA14 | CA13 | CA12
#define XROW3 CA11 | CA10 | CA09 | CA08

#define ZDIM_BOTTOM 0x10
#define ZDIM_TOP 0x80
#define ZDIM_TOPBOT 0x90

#define ZDIM0 0x10
#define ZDIM1 0x20
#define ZDIM2 0x40
#define ZDIM3 0x80

// a few patterns that can be used
#define CORNERS CA07 | CA11 | CA04 | CA08
#define OUTERRING XROW0 | XROW3 | YCOL0 | YCOL3
#define INNERRING CA01 | CA02 | CA13 | CA14

#define PATTERN_A CA04 | CA01 | CA14 | CA11
#define PATTERN_B YCOL2
#define PATTERN_C YCOL1
#define PATTERN_D XROW1
#define PATTERN_E XROW2
#define PATTERN_F CA07 | CA02 | CA13 | CA08
#define PATTERN_G INNERRING
#define PATTERN_H OUTERRING
#define PATTERN_I CORNERS

#define ZDIM_ALL 0xF0
#endif

watchdog.sh

This script actually shouldn’t be necessary at all. Occasionally the program stops. I had not been able to trace it down to either my libraries or my code, but in the end decided to simply write a small watchdog to ensure that if the program stops, it is restarted.

#!/bin/bash

PGM=/home/pi/picube/picube

while [ 1 == 1 ]
do

 CNT=`ps -ef | grep $PGM | grep -v grep | grep -v sudo | awk '{ print $8 }' | wc -l`
 NAME=`ps -ef | grep $PGM | grep -v grep | grep -v sudo | awk '{ print $8 }' `
 ID=`ps -ef | grep $PGM | grep -v grep | grep -v sudo | awk '{ print $2 }' `

 if [ $CNT -ne 1 ]
 then
 echo restarting
 $PGM &
 fi

 sleep 60
done

Makefile

PGM=picube

SRC=$(PGM).c
OBJ=$(SRC:.c=.o)
CFLAGS=-Wall
INCDIR=-I.
LIBS=-lbcm2835 -lm
LIBDIR=

all:    $(PGM)

$(PGM): $(OBJ) $(PGM).h
        @echo building $(PGM)
        gcc $(INCDIR) $(OBJ) $(LIBS) $(LIBDIR) -o $(PGM)

.c.o:
        @echo "compling $@"
        @gcc -c $(CFLAGS) $< -o $@

clean:
        @echo cleaning up build env
        @rm -f $(OBJ) $(PGM)

.phony: run
run:    all
        sudo ./$(PGM)

#dependancies
$(PGM).o: $(PGM.h)

Posted in programming, Setup From Scratch | Tagged c / c+, DIY | Comments Off

getting the most out of your computer – threading

Posted on September 19, 2016 by DocDock

Flashback to 2002

Technology is amazing. Pretty much since personal computers were created the speed of their processors have been increasing at almost a linear rate of doubling in speed every two years. This is awesome but it is also a physical impossibility that the technology will continue to decrease in size forever.

At the same time that the speed of the cpu’s is coming into question, the hardware companies have started to add multiple cores to their processors. The only trick is to be able to use them.

Back to the present

Processors are no longer doubling in speed every two years. Technology is progressing but now everything is more evolutionary than revolutionary. With one small exception. The CPU’s are packing in more cores.

A dual core machine was a noteworthy event a few years back but these days there is 64 bit quad core machines supporting one or two threads per core on every desk. Twelve core processors may not be on every desktop but for the price of a quad core machine from yesteryear, you can purchase a 12 core CPU today for the cost of the computer a few years back. Of course, then you will need to purchase the rest of the computer.

It is actually possible to create a big multi-core system with a reasonable budget rather than a princely budget.

With the acceleration of the CPU speeds slowing down the only way to get more power out of our machines is to effectively use all the processing power that is available.

Today the ability to take advantage of the underlying hardware is so much easier now than it has been in the past. The past is obviously when the IT guys were trying to take full advantage of machines with multiple single core processors.

Quicksort

Normal Java programs are just a regular single threaded processes. One of the main problems with multi-threaded programs is you need to have a problem that can easily be broken down into independent pieces.

I wanted to come up with a real world solution not just a couple of threads running independently sending some interleaved output to the console.

The only programming task that came to mind that was small enough to easily get an overview on was sorting a list- specifically the Quicksort method. The Quicksort method splits up the list into smaller sub lists while swapping items that are on the wrong side of the pivot spot. More information about Quicksort can be found in a lot computer science books or other web sites but a nice description of the Quicksort can also be found on wikipedia.

The sort is really efficient and much faster when compared to other sorting methods, but pretty much as expected it only uses one of the four cores that is available on my computer.

Quicksort code at bottom of page

Java helps out

Despite the bad wrap that Java has received over efficiency when compared to other compiled languages it does offer quite a lot to the developer right out of the box.

In this case, Java provides the Thread class which provides the infrastructure to allow anyone to run multiple tasks simultaneously, well to the best ability of the underlying hardware.

The Thread class is actually just another java object. There are a number of methods that are available to help control your thread.

start
run
join
isAlive
interrupt
sleep
yield

The run method is actually the implementation the interface Runnable. This is the actual starting point for the execution of the thread. What might seem a bit counter intuitive is that you don’t actually call this method yourself. If you call the run method directly then you are simply executing another method call from your program.

The actual starting of the thread is done by calling the start method. Java will then do all the necessary work to create a thread, either for your virtual machine or using a OS native thread.

The next and perhaps one of the most important methods is join. This method call will cause the calling thread, or the main program, to block until the called thread is finished.

The threaded Quicksort that I have developed actually doesn’t really need to check if the threads are alive due to how it was was programmed. The program simply creates threads as it recursively processes the data. If your problem were somewhat differently structured you might be starting a number of parallel threads and look at a later point if they have finished by using the isAlive method.

The last three methods on my list are really about controlling if a thread is running or should be blocked. The interrupt method is a way to force a thread to block. The yield method is almost the same thing. You are causing a thread to block itself to allow another waiting thread with the same priority to run.

The last method on the list is sleep. This is another way to block a thread, albeit in a slightly different manner. This method causes the thread to sleep for a given number of milliseconds. This at least tacitly implies that although it is giving someone else a shot at the system resources it will be run again after the sleep period is finished.

How threads are run and when they are blocked really depends on the scheduler. This would be dependent on both the operating system and Java virtual machine.

Converting over to threaded algorithm

The easiest way to take advantage of threading is to simply to create your own class to extend the Thread. When this is done the only thing the developer needs to implement is his own run method.

The neat thing about the Quicksort algorithm is that it is recursive. Sorting the entire list looks just like sorting any sub-part of the list.

The change to my sample quick sort program is simply to do each sub-list as a different thread. The code is actually hardly different at all, but does it actually work?

multi-threaded quicksort code at bottom of page

It works really well. This screenshot shows a spike on all four cores when I run the threaded version of the program.

single threaded
took 5495 milli seconds

multi threaded
took 2783 milli seconds

Why does this work ?

It is important to remember that Java passes it variables to methods by value. If this is the case, how can this sort routine work?

The thread example works because in Java variables are passed by value, but things are a bit special in the case of arrays. In this case, the value being passed is the reference to the array.

Yet when we manipulate the individual elements of the array those changes will be reflected at the source, so the passing of the array around “willy nilly” between threads works.

This entire solution also works because the Quicksort partitions the data into mutually exclusive data sets. This is why the program doesn’t have any semaphores to prevent us from corrupting our data by having two or more processes changing things simultaneously.

Finally it works because we are not changing what the “array” points to. If we did that, this would not be changed in the calling method as the array reference was passed by value. That would be exactly the same as passing in a integer to a method and changing it (while expecting it to change in the calling method)

It does four threads but can it do more? I ran the program on another computer that could run eight threads simultaneously and the program ran as expected. The cpu was saturated for a brief time. The threaded version, as expected, does run faster when provided with more CPU resources.

single threaded
took 4901 milli seconds

multi threaded
took 1318 milli seconds

If a little threading is good then a lot of threading must be better? Right?

The limit of threading is somewhat controlled in the Quicksort method. I only start a threaded version of the Quicksort if there are at least two million values in the list. This is because there is overhead associated with creating a thread.

Other important facts

The reason why this threaded application looks so simple is because it is definitely not production quality code. Just like memory or disk space there are limits to how many threads that can be created or run at one time.

The threaded version of the Quicksort doesn’t do any checks to ensure that it doesn’t over step the limits of the computer. The limits are dependent on a number of factors such as the operating system and the resources available.

Is this really worth the effort

Yes and no. Why don’t I have a lot of threaded programs laying about that can be used as examples. There are a number of reasons.

The problem must lend itself to being easily broken up
You must have enough data to make the task worthwhile

I needed a lot of data in order to keep my computer busy for five seconds utilizing a single thread. The flip side of the argument to using threads is that creating threads also take resources. If I create a thread for each recursion when using small data sets, the overall runtime is longer for the threaded version of the program. This extra run time is exclusively due to overhead for threads.

In order to get a good test, I created a small program to generate 40 million integers just to keep the CPU busy for a few seconds.

The threaded version of the program run about 3.6 seconds faster than the single threaded approach. If the additional development time to add only the threading was calculated to be one hour, the program might need to be run almost a thousand times or more to offset the extra development costs versus the single threaded version. This is might be fine for a task that runs every minute but perhaps a poor investment for a task that runs once a day.

Parting shot

Perhaps 30 years ago computer scientists got excited about sorting algorithms, but that was due to a lot of factors. Computing time was quite expensive and resources were very limited. There are a lot of things that are possible with a lot of cheap CPU, ram or disk space that would have made these old time developers cry about the inefficiency of some of the modern solutions.

Just for fun, I did a bubble sort and used the built in java sort to get some other numbers.

Java sort
55730 milliseconds

Bubble sort
***** milliseconds

Bubble sorting is known to be perhaps the easiest to understand for beginners but the most inefficient. The time required to sort 40 million integers was excessive and my lack of patients only lasted 2 1/2 days. I did some smaller runs which also ran fairly slowly, ten thousand integers were sorted in less than half a second while one million integers took about 70 minutes. The runtimes did not have a perfectly linear increase but based on some of the calculations it might take about 4 days to sort through ten million integers.

Parallel processing has to be part of the new reality and in the right situation it can indeed make a tremendous difference. Yet one of the real solutions is to use efficient algorithms, the right tool for the right situation and knowing when further optimizations are not adding value.

Note: It is quite unfair to the java sort to accuse it of taking such a long time to sort this data. The other sorts were using simple integers and a simple array. To use the build in sort it was necessary to use the Integer class and a collection. In any case, 55 seconds is probably a reasonable sort time on an old laptop for 40 million integers.

Code

Single threaded Quicksort

package singlethreaded;

/*
 * this example is an implementation of the quicksort
 * algorithm described in wikipedia entry.
 * 
 *      https://en.wikipedia.org/wiki/Quicksort
 * 
 */
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.ObjectInputStream;

public class Quicksort 
{
	int datalist[] = {  12, 16, 333, 50, 1000, 5, 897, 1, 3, 66, 13 };

	public void quicksort(int datatosort[], int lo, int hi) 
	{
		if (lo < hi)
		{
			int partidx = partition(datatosort, lo, hi);
			quicksort(datatosort, lo, partidx);
			quicksort(datatosort, partidx + 1, hi);
		}
	}

	/*
	 * Hoare partition scheme
	 */
	public int  partition(int datatosort[], int lo, int hi)
	{
		int pivot = datatosort[lo];
		int i = lo - 1;
		int j = hi + 1;

		while (true)
		{
			do {
				i = i + 1;
			} while (datatosort[i] < pivot); 

			do {
			        j = j - 1; 
			} while (datatosort[j] > pivot);

			if (i >= j) 
				break;

			swap(datatosort,i,j); 
		}
		return j;
	}

	void swap( int datatosort[], int left, int right)
	{
		int temp = datatosort[left];
		datatosort[left] = datatosort[right];
		datatosort[right] = temp;
	}

	public void load()
	{
		String filename = "testdata.bin";
		ObjectInputStream inputStream = null;
		try {
			inputStream = new ObjectInputStream(new FileInputStream(filename));
		} catch (FileNotFoundException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
		
		try {
			datalist = (int[])inputStream.readObject();
		} catch (ClassNotFoundException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
		
		try {
			inputStream.close();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}

	public void sort ()
	{
		quicksort(datalist,0,datalist.length - 1);
	}

	public static void main(String args[])
	{
		Quicksort sorter = new Quicksort();
		sorter.load();
		long nanoStart = System.nanoTime();
		sorter.sort();
		long nanoEnd = System.nanoTime();
		long nanoseconds = (nanoEnd - nanoStart);
		System.out.println("took " + nanoseconds + " nano seconds");
		System.out.println("took " + nanoseconds / 1000000 + " milli seconds");
	}
}

Threaded controller

package multithreaded;

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.ObjectInputStream;

public class Threadcontroller {

	int datalist[];
	
	public void sort ()
	{
		Threadquicksort sorter = new Threadquicksort();
		sorter.setupThread(datalist,0,datalist.length-1);
		sorter.start();


		// wait for it to finish
		try {
			sorter.join();
		} catch (InterruptedException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}

	public void load()
	{
		String filename = "testdata.bin";
		ObjectInputStream inputStream = null;
		try {
			inputStream = new ObjectInputStream(new FileInputStream(filename));
		} catch (FileNotFoundException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
		
		try {
			datalist = (int[])inputStream.readObject();
		} catch (ClassNotFoundException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
		
		try {
			inputStream.close();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}
	
	public void print(boolean small)
	{
		String output = "";
		
		if (small == true)
		{
			for (int idx = 0; idx < datalist.length; idx++)
				output = output + datalist[idx] + " ";

			System.out.println(output);
		}
		else
		{
			for (int idx = 0; idx < datalist.length; idx++)
				System.out.println(datalist[idx]);
		}
	}
	
	public static void main(String args[])
	{
		Threadcontroller controller = new Threadcontroller();

		controller.load();
		long nanoStart = System.nanoTime();
		controller.sort();
		long nanoEnd = System.nanoTime();
		long nanoseconds = (nanoEnd - nanoStart);
		System.out.println("took " + nanoseconds + " nano seconds");
		System.out.println("took " + nanoseconds / 1000000 + " milli seconds");
	}
}

Threaded sort

package multithreaded;

public class Threadquicksort extends Thread
{
	int referenceToDatalist[];
	int start, end;

	public void setupThread(int datalist[], int start, int end)
	{
		this.start = start;
		this.end = end;
		referenceToDatalist = datalist;
	}

	public void run()
	{
		quicksort(referenceToDatalist,start,end);
	}

	public void quicksort(int datatosort[], int lo, int hi) 
	{
		if (hi - lo < 2000000)
		{
			// there must be a border where creating threads costs more than it solves.
			if (lo < hi)
			{
				int partidx = partition(datatosort, lo, hi);
				quicksort(datatosort, lo, partidx);
				quicksort(datatosort, partidx + 1, hi);
			}
		}
		else
		{
			int partidx = partition(datatosort, lo, hi);
			Threadquicksort lowerhalf = new Threadquicksort();
			Threadquicksort upperhalf = new Threadquicksort();
			
			// tell them what to do
			lowerhalf.setupThread(datatosort, lo, partidx);
			upperhalf.setupThread(datatosort, partidx + 1, hi);
			
			// send them on their merry way
			lowerhalf.start();
			upperhalf.start();
			
			// wait for them to finish.
			try {
				lowerhalf.join();
			} catch (InterruptedException e) {
				// TODO Auto-generated catch block
				e.printStackTrace();
			}
			try {
				upperhalf.join();
			} catch (InterruptedException e) {
				// TODO Auto-generated catch block
				e.printStackTrace();
			}
		}
	}

	/*
	 * Hoare partition scheme
	 */
	public int  partition(int datatosort[], int lo, int hi)
	{
		int pivot = datatosort[lo];
		int i = lo - 1;
		int j = hi + 1;

		while (true)
		{
			do {
				i = i + 1;
			} while (datatosort[i] < pivot); 

			do {
			        j = j - 1; 
			} while (datatosort[j] > pivot);

			if (i >= j) 
				break;

			swap(datatosort,i,j); 
		}
		return j;
	}

	void swap( int datatosort[], int left, int right)
	{
		int temp = datatosort[left];
		datatosort[left] = datatosort[right];
		datatosort[right] = temp;
	}
}

Bubble sort

package singlethread;

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.ObjectInputStream;

public class Bubblesort 
{
	int datalist[] = {  12, 16, 333, 50, 1000, 5, 897, 1, 3, 66, 13 };

	public void bubblesort()
	{
		System.out.println("starting bubble sort for " + datalist.length + " items ");
		int totalswaps = 0;
		boolean swapped = false;
		
		do
		{
			swapped = false;
			for (int left = 0; left < datalist.length -1; left++) 
                        { 
                                int right = left + 1; if (datalist[left] > datalist[right])
				{
					int temp = datalist[left];
					datalist[left] = datalist[right];
					datalist[right] = temp;
					swapped = true;
					totalswaps++;
				}
			}	
		}
		while (swapped == true);

		System.out.println("total swaps " + totalswaps);
	}

	public void load()
	{
		String filename = "testdata.bin";
		ObjectInputStream inputStream = null;
		try {
			inputStream = new ObjectInputStream(new FileInputStream(filename));
		} catch (FileNotFoundException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
		
		try {
			datalist = (int[])inputStream.readObject();
		} catch (ClassNotFoundException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
		
		try {
			inputStream.close();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}

	public void print(boolean small)
	{
		String output = "";
		
		if (small == true)
		{
			for (int idx = 0; idx < datalist.length; idx++)
				output = output + datalist[idx] + " ";

			System.out.println(output);
		}
		else
		{
			for (int idx = 0; idx < datalist.length; idx++)
				System.out.println(datalist[idx]);
		}
	}

	public void test()
	{
		//Bubblesort sorter = new Bubblesort();

		load();
		
		long nanoStart = System.nanoTime();
		bubblesort();

		long nanoEnd = System.nanoTime();
		long nanoseconds = (nanoEnd - nanoStart);

		System.out.println("took " + nanoseconds + " nano seconds");
		System.out.println("took " + nanoseconds / 1000000 + " milli seconds");
		//print(false);
	}
	
	public static void main(String args[])
	{
		Bubblesort sorter = new Bubblesort();

		sorter.test();
	}
}

Java sort

package singlethreaded;
 
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Random;
 
public class Javasort {
 
       List datalist = new ArrayList();
      
       public void create(int len)
       {     
             Random rand = new Random();
            
             long nanoStart = System.nanoTime();
             for (int idx = 0; idx < len; idx++)
             {
                 // nextInt is normally exclusive of the top value,
                 // so add 1 to make it inclusive
                 int randomNum = rand.nextInt(10000001); // numbers 0 - 1000000
                 datalist.add(new Integer(randomNum));
 
                 //System.out.println(idx + ". " + randomNum);
             }
             long nanoEnd = System.nanoTime();
 
             long nanoseconds = (nanoEnd - nanoStart);
             System.out.println(nanoseconds);
       }
       public void sort()
       {
             Collections.sort(datalist);
       }
      
       public void print()
       {
             int i=0;
             for(Integer temp: datalist){
                    System.out.println("item " + ++i + " : " + temp);
             }
       }
      
       public static void main(String args[])
       {
             Javasort gendata = new Javasort();
             gendata.create(40000000);
            
             long nanoStart = System.nanoTime();
 
             gendata.sort();
 
             long nanoEnd = System.nanoTime();
             long nanoseconds = (nanoEnd - nanoStart);
 
             System.out.println("took " + nanoseconds + " nano seconds");
             System.out.println("took " + nanoseconds / 1000000 + " milli seconds");
       }
}

Generate sort data

package singlethreaded;
 
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.ObjectOutputStream;
import java.util.Random;
 
public class GenerateData {
 
                int datalist[] = null;
               
                public void create(int len)
                {             
                               datalist = new int[len];
                               Random rand = new Random();
                              
                               long nanoStart = System.nanoTime();
                               for (int idx = 0; idx < len; idx++)
                               {
                                   // nextInt is normally exclusive of the top value,
                                   // so add 1 to make it inclusive
                                   int randomNum = rand.nextInt(10000001); // numbers 0 - 1000000
                                   datalist[idx] = randomNum;
                                   //System.out.println(idx + ". " + randomNum);
                               }
                               long nanoEnd = System.nanoTime();
 
                               long nanoseconds = (nanoEnd - nanoStart);
                               System.out.println(nanoseconds);
                }
 
                public void save()
                {
                               String  filename = "testdata.bin";
                               ObjectOutputStream outputStream = null;
                              
                               try {
                                               outputStream = new ObjectOutputStream(new FileOutputStream(filename ));
                               } catch (FileNotFoundException e) {
                                               // TODO Auto-generated catch block
                                               e.printStackTrace();
                               } catch (IOException e) {
                                               // TODO Auto-generated catch block
                                               e.printStackTrace();
                               }
                              
                               try {
                                               outputStream.writeObject(datalist);
                               } catch (IOException e) {
                                               // TODO Auto-generated catch block
                                               e.printStackTrace();
                               }
                              
                               try {
                                               outputStream.close();
                               } catch (IOException e) {
                                               // TODO Auto-generated catch block
                                               e.printStackTrace();
                               }
                }
               
                public static void main(String args[])
                {
                               GenerateData gendata = new GenerateData();
                               gendata.create(40000000);
                               gendata.save();
                }
}

Ant build script

<project name="quicksort" default="world" basedir=".">

	<property file="project.properties"/>

	<property name="version.num" value="3.15.047"/>
	<property name="build.num" value="6"/>
	<property name="mainclass" value="singlethreaded.Quicksort"/>
	<property name="lib.dir" value="${basedir}/libs"/>
	<property name="src.dir" value="${basedir}/src"/>
	<property name="build.dir" value="${basedir}/bin"/>
	<property name="jar.dir" value="${basedir}/final"/>

	<path id="classpath">
		<fileset dir="${lib.dir}" includes="**/*.jar"/>
	</path>
 
	<target name="clean" description="remove anything we created">
		<echo>"cleaning ..."</echo>
		<delete dir="${class.dir}"/>
		<delete dir="${build.dir}"/>
	</target>

	<target name="compile" description="compile source code">
		<echo>"compiling ..."</echo>
		<mkdir dir="${build.dir}"/>
		<javac srcdir="${src.dir}" destdir="${build.dir}" classpathref="classpath" />
	</target>
 
	<target name="jarfile" depends="compile" description="generate jar file">
		<echo>"build jar ..."</echo>
		<mkdir dir="${jar.dir}"/>
 
		<tstamp>
			<format property="today.timestamp" pattern="yyyy-MM-dd HH:mm:ss" />
		</tstamp>
 
		<jar destfile="${jar.dir}/${jarname}.jar" basedir="${build.dir}">
			<manifest>
			<attribute name="Built-By" value="${user.name}" />
			<attribute name="Main-Class" value="${mainclass}" />
			<attribute name="Implementation-Version" value="${version.num} build #${build.num}"/>
			<attribute name="Built-Date" value="${today.timestamp}"/>
			</manifest>
		</jar>
	</target>
 
	<target name="world" depends="jarfile">
		<echo>"build script version 1.00"</echo>
	</target>
 
</project>

Posted in programming | Tagged best practices, java, programming | Comments Off

computer programs in the 21st century

Posted on September 16, 2016 by DocDock

The app I downloaded just doesn’t work. I am missing a lot of the photos that I took.

Great! I am at home and it looks like work has somehow followed me.

That great application that my wife was bragging about the previous day was now causing her great distress. The four hours she spent photographing created an incomplete list of files.

The application allows you to use your camera to take photos of photos in your physical photo album. The application crops the photo and uploads them to their server. You can then download the photos to your laptop.

The first day every photo showed up online annotated with all of her notes. The second day approximately half of the photos that she took were missing. She wasn’t sure which photos went missing but it might have been the photos from the end of her photo session.

A few things popped into my head. Is it possible that this free service only shows the first 500 photos. Nope, that wasn’t it.

She downloaded everything and deleted everything and tried again. This time some files were showing up while others were missing. The harder she tried the more photos were missing.

I gave it some thought and the facts of the problem was that the photographs were not being stored in the normal photo album. Not the ones that were lost nor the ones that were successful. I watched and could see that she was taking more than one photo a minute.

My idea was perhaps these “temporary files” are being stored elsewhere but rather than using the built in naming convention the app developers decided to be clever. Maybe they were taking their own naive approach to naming them. Maybe they used the same name for the current ph0to?

Sure enough, when my wife took no more than a single photo a minute then all of a sudden everything worked perfectly. No more pictures got lost.

The wife is once again happy. The key is to think like a programmer, the good ones not the lazy ones.

Posted in Soapbox | Tagged review | 1 Comment

“lp” and the automation of printing PDF’s

Posted on September 13, 2016 by DocDock

Apache FOP is actually really neat. With a few jar files, an xslt file and a xml file it is possible to create PDF file which can then be viewed or printed on any PC without any loss of formatting. It is so easy to write programs or scripts to create PDF files, so many you might want a more automated method for printing them.

I looked at my PDF viewer to see if it was possible to run it in a batch mode with a few parameters to automate the printing process of the PDF’s. I had been hoping a well placed command line argument would do it but I couldn’t really find anything helpful.

Google was able to find some hints on how I could easily do that on Linux (sorry windows users) without a normal PDF viewer.

The line printer command, lp, is used for the general printing of files. It can also be used to print PDF files as well. Like some Linux commands there are quite a few command line options.

LP – line printer

Option	Description	Example
-d <dest>	The name of the printer to use.	-d “HP-Color-LaserJet-CM1312-MFP”
-n <copies>	How many copies of document. Three pages in this example.	-n 3
-t <job name>	Sets the job name.	-t “birthday list”
-n <copies>	How many copies of document. Three pages in this example.	-n 3
-o <name=value>	Pass common job options to line printer program.
-o number-up=<count>	This will print multiple document pages to the printed page. Two in this case.	-o number-up=2
-o <media=size>	This will allow the document to be printed to different sizes of paper. The options are A4, letter and legal.	-o media=A4
-o <landscape>	This will rotate the document 90 degrees.	-o landscape
-o <landscape>	Pass common job options to line printer program.	-o number-up=2

For a full list of the command line options take a quick look at the man page for lp.

The simplest case is to simply name the printer and the file.

lp -d “HP-Color-LaserJet-CM1312-MFP” totals.pdf

Printing is usually a very dull command as it normally just works. If for some reason you need to check up on it there are a few ways this can be done. One of the most visible ways to see the printers would be to enter the following URL.

http://localhost:631/printers

Note: This assumes that cups is installed. If for some reason cups is not installed just install using apt-get

sudo apt-get update && sudo apt-get install cups cups-client

This is probably more interesting when the number of printers is greater than one. To see the contents, simply click on the printer to bring up the actual contents of the print queue.

The easiest way to remove any jobs from the print queue is from this screen. Press the cancel job button and you will be prompted for the username and password, after which the job is deleted.

Of course, there are command line programs which could be used as well.

LPQ – line printer queue

This command will display a list of all the jobs that are currently waiting to print.

Option	Description	Example
-P <dest>	The name of the printer to use.	-P “HP-Color-LaserJet-CM1312-MFP”

If your printer is the default printer you don’t need to give name name of the printer to the command. The default print queue will be displayed.

LPRM – cancel a print job

The lprm command will delete an entry from your print queue.

Option	Description	Example
-P <dest>	The name of the printer to use.	-P “HP-Color-LaserJet-CM1312-MFP”

Deleting print jobs are easily done from the command line as well.

gast@asus:~$ lpq -P HP-Color-LaserJet-CM1312-MFP
HP-Color-LaserJet-CM1312-MFP is ready
Rank Owner Job File(s) Total Size
1st dock 24 YearlyAverageCurrencyExchangeRa 1429504 bytes
gast@asus:~$ 
gast@asus:~$ lprm -P "HP-Color-LaserJet-CM1312-MFP" 24
gast@asus:~$ 
gast@asus:~$ lpq
HP-Color-LaserJet-CM1312-MFP is ready
no entries

References

http://www.tutorialspoint.com/unix_commands/lp.htm

http://www.tutorialspoint.com/unix_commands/lpq-cups.htm

http://www.tutorialspoint.com/unix_commands/lprm.htm

Posted in Command line | Tagged linux | Comments Off

Security through obscurity – pdf removing passwords

Posted on September 7, 2016 by DocDock

Initially I was not planning on writing about how to crack pdf passwords. After all, that would be irresponsible much in the way giving a knife to a small child is irresponsible – right?

Well, simply typing in the following three words into your search engine.

pdf password recovery

This search will return quite a few entries that can be purchased or downloaded for free. This blog posting will help you most with pdf’s you created with a password that wasn’t overly strong. If you are looking strong passwords you might be taking months or centuries to crack your password.

I received a pdf file that was protected with a password. Apparently there was a slight miscommunication of what the password was and so I couldn’t open the file that was intended for me.

Guessing passwords, especially if you have no idea about the length or how secure the password is – is foolish. Despite how fast the computers are today they are not fast enough to break an arbitrarily long and complex password.

A four character password actually can be brute force broken without any real effort in less than 2 minutes

>echo 23:31:42,30
23:31:42,30

>..\pdfcrack file-4char.pdf
PDF version 1.4
Security Handler: Standard
V: 2
R: 3
P: -3904
Length: 128
Encrypted Metadata: True
FileID: ee806bec8c858dd1bdd1575322c0202b
U: 5a1e5f56878012e4b920c42d2616f85900000000000000000000000000000000
O: 18512d0db798a3154a508336074c85f9171b9147980a8a2a48009fc1a1cf91aa
Average Speed: 42185.4 w/s. Current Word: ‘lDGc’
Average Speed: 42201.3 w/s. Current Word: ‘Gceg’
Average Speed: 42447.3 w/s. Current Word: ‘m3Mj’
Average Speed: 42391.4 w/s. Current Word: ‘1Bln’
Average Speed: 42208.1 w/s. Current Word: ‘xdTq’
found user-password: ‘Pass’

>echo 23:33:31,21
23:33:31,21

If you can narrow down the composition of the password it is even faster at about a minute.

>echo 22:56:23,41
22:56:23,41

>..\pdfcrack file-4char.pdf -c abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ -n 4
PDF version 1.4
Security Handler: Standard
V: 2
R: 3
P: -3904
Length: 128
Encrypted Metadata: True
FileID: ee806bec8c858dd1bdd1575322c0202b
U: 5a1e5f56878012e4b920c42d2616f85900000000000000000000000000000000
O: 18512d0db798a3154a508336074c85f9171b9147980a8a2a48009fc1a1cf91aa
Average Speed: 42185.2 w/s. Current Word: ‘ebag’
Average Speed: 42079.4 w/s. Current Word: ‘znZl’
Average Speed: 42294.0 w/s. Current Word: ‘veas’
found user-password: ‘Pass’

echo 22:57:24,61
22:57:24,61

Just extending the password by one more character moves the task from the minutes to the hours.

I was lucky because the document wasn’t that important and the password was all lowercase letters (and not too long).

Had the task been a bit more complex I never could have managed to crack the code that fast. Had it taken longer, the time from that timezone would have changed and I would have been able to pick up the phone and get a new file.

The pdfcrack utility doesn’t take full advantage of multi threading on modern computers. This is actually the only serious drawback for this program. The task of brute forcing passwords while using it is less of a slam dunk with one thread versus four or more.

pdfcrack -t 4 receipe128.pdf

1:13:24,60
PDF version 1.4
Security Handler: Standard
V: 2
R: 3
P: -1852
Length: 128
Encrypted Metadata: True
FileID: 2625660821a90493e6068fab2aae37c8
U: 4e25fb3aea8995c47eed5bcafb67bc0300000000000000000000000000000000
O: 793706a064e35128a24bb79929a29805a1b0e20163d8edcf431da912648bdcae
thread 0 started
thread 1 started
thread 2 started
thread 3 started
Average Speed: 161666.2 w/s. Current Word: ‘3Jxc’
Average Speed: 161633.8 w/s. Current Word: ‘7iWf’
Average Speed: 162077.2 w/s. Current Word: ‘zslj’
Average Speed: 162074.8 w/s. Current Word: ‘5DKm’
Thread 3 found password: ‘Pass’
joined thread 0
joined thread 1
joined thread 2
joined thread 3
found user-password: ‘Pass’
1:13:53,10

The changes to support multi threading don’t appear to be base build of the pdfcrack tool but it is possible to get a copy that does from the following site.

http://andi.flowrider.ch/research/pdfcrack.html

Source code

http://andi.flowrider.ch/research/public/pdfcrack/pdfcrack.tar

Windows binary

http://andi.flowrider.ch/research/public/pdfcrack/pdfcrack_windows.zip

During my testing I had a chance to experiment more with PDF’s. It is a lot easier during testing if you can easily add and remove passwords.

The PDFTK tool does offer a number of different abilities to manipulate pdf files including both adding and removing passwords.

Secure PDF with password

The entire reason I became familiar with the PDFTK tool was because it was a command line program that could be used to encrypt pdf files. It is much easier to quickly type up the following command to “lock up” your pdf.

pdftk <source pdf> output <encrypted pdf> owner_pw <owner pass> user_pw <user pass> allow printing

pdftk receipe.pdf output receipe-encr128.pdf owner_pw secretpw user_pw pass6

The owner password is required for setting or clearing the password to prevent people from reading your document. The user password is that password. It is the user password that you are prompted for when opening up your encrypted PDF file.

The last two parameters on the command line actually are very self explanatory. When you add “allow printing” you can also print the PDF file once you enter your password. It is not clear if every open source PDF viewer respects that flag but Adobe reader certainly does.

There are a few other permissions you can grant.

DegradedPrinting
ModifyContents
Assembly
CopyContents
ScreenReaders
ModifyAnnotations
FillIn
AllFeatures

If you want to be able to print out your encrypted file, don’t forget to add “allow printing”.

Remove PDF password

Removing a password is just as easy as adding it. Simply pick your file and provide your owner password and it will be removed.

pdftk <source pdf> input_pw <pass> output <destination pdf>

pdftk receipe128.pdf input_pw secretpw output receipe-nopass.pdf

Extract a single page

The PDFTK tool can also manipulate the pdf itself not just deal with the security.

The PDFTK tool works great at extracting one or more pages from a larger pdf to create a smaller pdf. Whats more, the command actually seems pretty clear what it needs and what it is doing.

pdftk <source pdf> cat <page or range> output <destination pdf> 

pdftk main.pdf cat 1 output page1.pdf

pdftk main.pdf cat 1-8 output firsteightpages.pdf

This example is creating a new pdf called page1.pdf which will contain page 1 from our pdf called “main.pdf”. It isn’t clear why the parameter “cat” wasn’t perhaps page or range to be a bit more friendly to the people less comfortable with the command line. Yet in general, most commands are a bit arbitrary and you simply need to memorize the name, their parameters and what the command does does.

The PDFTK program does the extraction with no output to the screen. If you want a bit more feedback from the program simply add the parameter “verbose” at the end. The output doesn’t provide much that is interesting.

It is usually easier to run the command and if you do make a mistake the error will be displayed.

Extract all pages

After seeing what is necessary to extract a single page, my mind had already created a shell script to go through the process and extract all pages.

This must be a fairly common desire when manipulating pdfs as there is a parameter for doing this.

pdftk <source pdf> burst

pdftk main.pdf burst

PDFTK will create as many pg_####.pdf files as there are pages.

ls -ltr
total 328
drwxr-xr-x 2 cdock cdock 4096 Mar 11 2015 Videos
drwxr-xr-x 2 cdock cdock 4096 Mar 11 2015 Templates
drwxr-xr-x 2 cdock cdock 4096 Mar 11 2015 Public
drwxr-xr-x 2 cdock cdock 4096 Mar 11 2015 Pictures
drwxr-xr-x 2 cdock cdock 4096 Mar 11 2015 Music
drwxr-xr-x 2 cdock cdock 4096 Mar 11 2015 Downloads
drwxr-xr-x 2 cdock cdock 4096 Mar 11 2015 Documents
drwxr-xr-x 2 cdock cdock 4096 Mar 11 2015 Desktop
-rw-r–r– 1 cdock cdock 69845 Sep 6 16:01 main.pdf

> pdftk main.pdf burst
> ls -ltr
total 484
drwxr-xr-x 2 cdock cdock 4096 Mar 11 2015 Videos
drwxr-xr-x 2 cdock cdock 4096 Mar 11 2015 Templates
drwxr-xr-x 2 cdock cdock 4096 Mar 11 2015 Public
drwxr-xr-x 2 cdock cdock 4096 Mar 11 2015 Pictures
drwxr-xr-x 2 cdock cdock 4096 Mar 11 2015 Music
drwxr-xr-x 2 cdock cdock 4096 Mar 11 2015 Downloads
drwxr-xr-x 2 cdock cdock 4096 Mar 11 2015 Documents
drwxr-xr-x 2 cdock cdock 4096 Mar 11 2015 Desktop
-rw-r–r– 1 cdock cdock 69845 Sep 6 16:01 main.pdf
-rw-r–r– 1 cdock cdock 43184 Sep 6 16:55 pg_0002.pdf
-rw-r–r– 1 cdock cdock 52053 Sep 6 16:55 pg_0001.pdf
-rw-r–r– 1 cdock cdock 54452 Sep 6 16:55 pg_0003.pdf
-rw-r–r– 1 cdock cdock 505 Sep 6 16:55 doc_data.txt

The final file create, doc_data.txt, contains information about the pdf itself.

Assembly of new PDF’s

Merging multiple pdf files into a single pdf is also trivial. Simply list all of them on the command line and they are placed into the final pdf file in the same order they are listed on the command line.

pdftk <source pdf 1> <source pdf 2> ... <source pdf n>  cat output newfile.pdf
pdftk pg_0001.pdf pg_0002.pdf pg_0003.pdf cat output newfile.pdf

Installing Windows software

The good news is that even if you aren’t a Linux person you can download windows versions of these utilities.

PDFCrack Windows binary

http://andi.flowrider.ch/research/public/pdfcrack/pdfcrack_windows.zip

PDFTK Windows binary

https://www.pdflabs.com/tools/pdftk-the-pdf-toolkit/

Installing Linux software

Installation on Linux may vary depending on your distribution. On the Debian based distros simply use apt-get to get your own copy of either pdftk or pdfcrack.

Install PDFCrack

> sudo apt-get install pdfcrack
[sudo] password for dock:
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following packages were automatically installed and are no longer required:
gstreamer1.0-pulseaudio icedtea-netx-common libasn1-8-heimdal libatk-wrapper-java libatk-wrapper-java-jni libfreerdp-rail1.1
libgssapi3-heimdal libhcrypto4-heimdal libheimbase1-heimdal libheimntlm0-heimdal libhx509-5-heimdal libkrb5-26-heimdal
liblivemedia23 libmpg123-0 libpostproc52 libproxy-tools libroken18-heimdal libusageenvironment1 libwind0-heimdal
Use ‘apt-get autoremove’ to remove them.
The following NEW packages will be installed:
pdfcrack
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 32.4 kB of archives.
After this operation, 105 kB of additional disk space will be used.
Get:1 http://ftp.de.debian.org/debian/ jessie/main pdfcrack amd64 0.14-2 [32.4 kB]
Fetched 32.4 kB in 0s (167 kB/s)
Selecting previously unselected package pdfcrack.
(Reading database … 144680 files and directories currently installed.)
Preparing to unpack …/pdfcrack_0.14-2_amd64.deb …
Unpacking pdfcrack (0.14-2) …
Processing triggers for man-db (2.7.0.2-5) …
Setting up pdfcrack (0.14-2) …

Install PDFTK

> sudo apt-get install pdftk
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following packages were automatically installed and are no longer required:
gstreamer1.0-pulseaudio icedtea-netx-common libasn1-8-heimdal libatk-wrapper-java libatk-wrapper-java-jni libfreerdp-rail1.1
libgssapi3-heimdal libhcrypto4-heimdal libheimbase1-heimdal libheimntlm0-heimdal libhx509-5-heimdal libkrb5-26-heimdal
liblivemedia23 libmpg123-0 libpostproc52 libproxy-tools libroken18-heimdal libusageenvironment1 libwind0-heimdal
Use ‘apt-get autoremove’ to remove them.
The following extra packages will be installed:
gcj-4.9-jre-lib libgcj-common libgcj15
Suggested packages:
libgcj15-dbg libgcj15-awt
The following NEW packages will be installed:
gcj-4.9-jre-lib libgcj-common libgcj15 pdftk
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 20.4 MB of archives.
After this operation, 70.6 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://ftp.de.debian.org/debian/ jessie/main libgcj-common all 1:4.9.1-7 [160 kB]
Get:2 http://ftp.de.debian.org/debian/ jessie/main libgcj15 amd64 4.9.2-10 [9,224 kB]
Get:3 http://ftp.de.debian.org/debian/ jessie/main gcj-4.9-jre-lib all 4.9.2-10 [10.3 MB]
Get:4 http://ftp.de.debian.org/debian/ jessie/main pdftk amd64 2.02-2 [678 kB]
Fetched 20.4 MB in 23s (851 kB/s)
Selecting previously unselected package libgcj-common.
(Reading database … 144687 files and directories currently installed.)
Preparing to unpack …/libgcj-common_1%3a4.9.1-7_all.deb …
Unpacking libgcj-common (1:4.9.1-7) …
Selecting previously unselected package libgcj15:amd64.
Preparing to unpack …/libgcj15_4.9.2-10_amd64.deb …
Unpacking libgcj15:amd64 (4.9.2-10) …
Selecting previously unselected package gcj-4.9-jre-lib.
Preparing to unpack …/gcj-4.9-jre-lib_4.9.2-10_all.deb …
Unpacking gcj-4.9-jre-lib (4.9.2-10) …
Selecting previously unselected package pdftk.
Preparing to unpack …/pdftk_2.02-2_amd64.deb …
Unpacking pdftk (2.02-2) …
Processing triggers for man-db (2.7.0.2-5) …
Setting up libgcj-common (1:4.9.1-7) …
Setting up libgcj15:amd64 (4.9.2-10) …
Setting up gcj-4.9-jre-lib (4.9.2-10) …
Setting up pdftk (2.02-2) …
Processing triggers for libc-bin (2.19-18+deb8u4) …

Posted in Command line, security, Setup From Scratch | Comments Off

better logging than printf – using log4j – part III

Posted on September 4, 2016 by DocDock

Log4j is a very convenient tool to have in your programming arsenal. This is true even if only the very basic functionality is being used. The basic functionality of log4j2 was covered in my previous example.

To be honest, the standard levels are truly good enough for most applications but the neat feature of version 2 is the ability to create custom output levels.

In order to create some new levels we do need to know how the existing levels relate to each other.

Standard Level	Integer value
OFF	0
FATAL	100
ERROR	200
WARN	300
INFO	400
DEBUG	500
TRACE	600
ALL	Integer.MAX_VALUE

From looking at these levels there is quite a bit of space between each integer value for a new level to be slipped in. These levels could be developer or site specific. They could even be used to help track information for other internal needs (ie auditing).

To provide an example of this, I will create two new levels, one being to generate a silly amount of debug information while the other would be to capture auditing information.

Custom Levels

Actually very little extra work is necessary to create new levels. Simply use the new Level class to create a new level. Simply decide where in the range of values the value should fit in.

Create new level

Level AUDIT = Level.forName("AUDIT",350);
Level SPECIAL = Level.forName("SPECIAL", 399);

Apache also provides us with a new method on the Logger to call when using the new level. The method “log” accepts the new level as the first parameter followed by the data to be logged.

Using new level

logfile.log(AUDIT, "first auditing message");
logfile.log(AUDIT, "second auditing message");

There is really not a lot of special handling that is necessary when using these new levels but it is important that the new log levels are created before retrieving the logger from the LogManager.

Examining the output

The full program and configuration file listing is at the bottom. What is more interesting is looking at some of the output that would be generated by modifying the log4j2.xml file especially when we change the root level.

The most basic setting is to just use the normal info level.

<Root level=”info”>
<AppenderRef ref=”STDOUT” />
<AppenderRef ref=”FILE” />
</Root>

We can see in our log output that one of our new debug levels, AUDIT, is now showing up in the middle of our log output. This might not be appropriate for real auditing but might be useful for some sort of diagnostics for special hardware or special situations.

2016-09-02 21:22:56 | INFO  | utility.CustomVersion (CustomVersion.java:18) - Program starting at 
2016-09-02 21:22:56 | AUDIT | utility.CustomVersion (CustomVersion.java:21) - first auditing message
2016-09-02 21:22:56 | AUDIT | utility.CustomVersion (CustomVersion.java:22) - second auditing message
2016-09-02 21:22:56 | INFO  | utility.CustomVersion (CustomVersion.java:28) - 1 x 1 = 1
2016-09-02 21:22:56 | INFO  | utility.CustomVersion (CustomVersion.java:28) - 2 x 2 = 4
2016-09-02 21:22:56 | INFO  | utility.CustomVersion (CustomVersion.java:28) - 3 x 3 = 9
2016-09-02 21:22:56 | AUDIT | utility.CustomMathV (CustomMathV.java:30) - extremely large numbers being used in math - value = '4'
2016-09-02 21:22:56 | INFO  | utility.CustomVersion (CustomVersion.java:28) - 4 x 4 = 16
2016-09-02 21:22:56 | AUDIT | utility.CustomMathV (CustomMathV.java:30) - extremely large numbers being used in math - value = '5'
2016-09-02 21:22:56 | INFO  | utility.CustomVersion (CustomVersion.java:28) - 5 x 5 = 25
2016-09-02 21:22:56 | INFO  | utility.CustomVersion (CustomVersion.java:40) - Program finishing at

The most interesting change to the output was when setting the level in our configuration file to our new custom level – “audit”. This actually performed as a filter for that level and only that level in the the output.

This was pretty neat as by default log4j2 doesn’t know anything about our new level when it reads in the configuration file.

<Root level=”audit”>
<AppenderRef ref=”STDOUT” />
<AppenderRef ref=”FILE” />
</Root>

2016-09-02 21:24:23 | AUDIT | utility.CustomVersion (CustomVersion.java:21) - first auditing message
2016-09-02 21:24:23 | AUDIT | utility.CustomVersion (CustomVersion.java:22) - second auditing message
2016-09-02 21:24:23 | AUDIT | utility.CustomMathV (CustomMathV.java:30) - extremely large numbers being used in math - value = '4'
2016-09-02 21:24:23 | AUDIT | utility.CustomMathV (CustomMathV.java:30) - extremely large numbers being used in math - value = '5'

Finally when changing the filter level to all we will see all log output including our two new levels. You can see both AUDIT and the SILLYDEBUG levels as expected.

<Root level=”all”>
<AppenderRef ref=”STDOUT” />
<AppenderRef ref=”FILE” />
</Root>

2016-09-02 21:25:09 | INFO  | utility.CustomVersion (CustomVersion.java:18) - Program starting at                                                                                                                                                                              
2016-09-02 21:25:09 | TRACE | utility.CustomVersion (CustomVersion.java:19) - add a trace to log output                                                                                                                                                                        
2016-09-02 21:25:09 | AUDIT | utility.CustomVersion (CustomVersion.java:21) - first auditing message                                                                                                                                                                           
2016-09-02 21:25:09 | AUDIT | utility.CustomVersion (CustomVersion.java:22) - second auditing message                                                                                                                                                                          
2016-09-02 21:25:09 | SILLYDEBUG | utility.CustomMathV (CustomMathV.java:20) - entering square                                                                                                                                                                                 
2016-09-02 21:25:09 | DEBUG | utility.CustomMathV (CustomMathV.java:21) - parameter is 1                                                                                                                                                                                       
2016-09-02 21:25:09 | SILLYDEBUG | utility.CustomMathV (CustomMathV.java:23) - mathmatics of squared using 1                                                                                                                                                                   
2016-09-02 21:25:09 | SILLYDEBUG | utility.CustomMathV (CustomMathV.java:37) - exiting square                                                                                                                                                                                  
2016-09-02 21:25:09 | INFO  | utility.CustomVersion (CustomVersion.java:28) - 1 x 1 = 1                                                                                                                                                                                        
2016-09-02 21:25:09 | SILLYDEBUG | utility.CustomMathV (CustomMathV.java:20) - entering square                                                                                                                                                                                 
2016-09-02 21:25:09 | DEBUG | utility.CustomMathV (CustomMathV.java:21) - parameter is 2                                                                                                                                                                                       
2016-09-02 21:25:09 | SILLYDEBUG | utility.CustomMathV (CustomMathV.java:23) - mathmatics of squared using 2                                                                                                                                                                   
2016-09-02 21:25:09 | SILLYDEBUG | utility.CustomMathV (CustomMathV.java:37) - exiting square                                                                                                                                                                                  
2016-09-02 21:25:09 | INFO  | utility.CustomVersion (CustomVersion.java:28) - 2 x 2 = 4                                                                                                                                                                                        
2016-09-02 21:25:09 | SILLYDEBUG | utility.CustomMathV (CustomMathV.java:20) - entering square                                                                                                                                                                                 
2016-09-02 21:25:09 | DEBUG | utility.CustomMathV (CustomMathV.java:21) - parameter is 3                                                                                                                                                                                       
2016-09-02 21:25:09 | SILLYDEBUG | utility.CustomMathV (CustomMathV.java:23) - mathmatics of squared using 3                                                                                                                                                                   
2016-09-02 21:25:09 | SILLYDEBUG | utility.CustomMathV (CustomMathV.java:37) - exiting square                                                                                                                                                                                  
2016-09-02 21:25:09 | INFO  | utility.CustomVersion (CustomVersion.java:28) - 3 x 3 = 9                                                                                                                                                                                        
2016-09-02 21:25:09 | SILLYDEBUG | utility.CustomMathV (CustomMathV.java:20) - entering square                                                                                                                                                                                 
2016-09-02 21:25:09 | DEBUG | utility.CustomMathV (CustomMathV.java:21) - parameter is 4                                                                                                                                                                                       
2016-09-02 21:25:09 | SILLYDEBUG | utility.CustomMathV (CustomMathV.java:23) - mathmatics of squared using 4                                                                                                                                                                   
2016-09-02 21:25:09 | AUDIT | utility.CustomMathV (CustomMathV.java:30) - extremely large numbers being used in math - value = '4'                                                                                                                                             
2016-09-02 21:25:09 | SILLYDEBUG | utility.CustomMathV (CustomMathV.java:37) - exiting square                                                                                                                                                                                  
2016-09-02 21:25:09 | INFO  | utility.CustomVersion (CustomVersion.java:28) - 4 x 4 = 16                                                                                                                                                                                       
2016-09-02 21:25:09 | SILLYDEBUG | utility.CustomMathV (CustomMathV.java:20) - entering square                                                                                                                                                                                 
2016-09-02 21:25:09 | DEBUG | utility.CustomMathV (CustomMathV.java:21) - parameter is 5                                                                                                                                                                                       
2016-09-02 21:25:09 | SILLYDEBUG | utility.CustomMathV (CustomMathV.java:23) - mathmatics of squared using 5                                                                                                                                                                   
2016-09-02 21:25:09 | AUDIT | utility.CustomMathV (CustomMathV.java:30) - extremely large numbers being used in math - value = '5'                                                                                                                                             
2016-09-02 21:25:09 | SILLYDEBUG | utility.CustomMathV (CustomMathV.java:37) - exiting square                                                                                                                                                                                  
2016-09-02 21:25:09 | INFO  | utility.CustomVersion (CustomVersion.java:28) - 5 x 5 = 25                                                                                                                                                                                       
2016-09-02 21:25:09 | DEBUG | utility.CustomVersion (CustomVersion.java:39) - add some additional debug to output                                                                                                                                                              
2016-09-02 21:25:09 | INFO  | utility.CustomVersion (CustomVersion.java:40) - Program finishing at

Other discoveries

One rather interesting thing about these custom levels was how forgiving log4j2 seemed to be of levels that did not exist.

During my testing I accidentally created an unknown level due to a typo only to discover that that message was silently ignored. If I then created the level prior to it being used, the level functioned as exactly as it should. It is not clear if this is a bug or a feature.

I made another interesting discovery while fooling around with Apache Ant. It was surprising not to see the file and line number in my log output. It took a few minutes of head scratching to realize that the program was not compiled with debugging on.

DEBUG | 2016-08-30 23:08:56 | [main] de.companyname.helloworld.HelloWorld (null:-1) - HelloWorld starting
INFO  | 2016-08-30 23:08:56 | [main] de.companyname.helloworld.HelloWorld (null:-1) - Hello World!
DEBUG | 2016-08-30 23:08:56 | [main] de.companyname.helloworld.HelloWorld (null:-1) - HelloWorld finishing

The solution was easy enough. Simply change the ant script to turn on debugging while compiling and recompile the program.

Test program

CustomProgram.java

package de.companyname.utility;


import org.apache.logging.log4j.Level;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;


public class CustomVersion {

	static Level AUDIT = Level.forName("AUDIT",350);
	static Level SPECIAL = Level.forName("SPECIAL", 399);
	
	public static void main(String[] args) 
	{
		Logger logfile = LogManager.getLogger(CustomVersion.class.getName());
		
		logfile.info("Program starting at " + "");
		logfile.trace("add a trace to log output");
		
		logfile.log(AUDIT, "first auditing message");
		logfile.log(AUDIT, "second auditing message");
		
		for (int index = 1; index <= 5; index++)
		{
			try {
				int square = de.companyname.utility.CustomMathV.square(index);
				logfile.info(index + " x " + index + " = " + square);
			}
			catch (Exception ex)
			{
				logfile.fatal(" while processing value " + index);
				logfile.fatal(ex.getMessage());
			}
		}

		logfile.log(Level.getLevel("UNDEFINED"), "some xxxx message"); 

		logfile.debug("add some additional debug to output");
		logfile.info("Program finishing at " + "");
	}
}

CustomMathV.java

package de.companyname.utility;


import org.apache.logging.log4j.Level;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;

public class CustomMathV {

	/*
	 * will return the square of the value input.
	 * this method will only be for the range of positive numbers.
	 */
	static public int square (int value) throws Exception
	{
		Level AUDIT = Level.forName("AUDIT", 350);
		Level SILLYDEBUG = Level.forName("SILLYDEBUG", 9999); 
		Logger logfile = LogManager.getLogger(CustomMathV.class.getName());
		
		logfile.log(SILLYDEBUG, "entering square");
		logfile.debug("parameter is " + value);
		
		logfile.log(SILLYDEBUG,  "mathmatics of squared using " + value);
		int retval = 0;
		
		if (value > 0)
		{
			retval = value * value;
			if (value >= 4)
				logfile.log(AUDIT,"extremely large numbers being used in math - value = '" + value + "'");
		}
		else
		{
			logfile.debug("invalid value of " + value + " given");
		}
		
		logfile.log(SILLYDEBUG, "exiting square");
		return retval;
	}
}

log4j2-custom.xml

<?xml version="1.0" encoding="UTF-8"?>
<Configuration xmlns="http://logging.apache.org/log4j/2.0/config">

	<Properties>
		<Property name="LOG_NAME">logfile.log</Property>
		<Property name="FILEPATTERN">%-5p | %d{yyyy-MM-dd HH:mm:ss} | [%t] %C (%F:%L) - %m%n</Property>
		<Property name="PATTERN">%d{yyyy-MM-dd HH:mm:ss} | %-5p | %C{2} (%F:%L) - %m%n</Property>
	</Properties>

	<Appenders>
		<File name="FILE" fileName="${LOG_NAME}" append="true">
			<PatternLayout pattern="${FILEPATTERN}"/>
		</File>
		<Console name="STDOUT" target="SYSTEM_OUT">
			<PatternLayout pattern="${PATTERN}"/>
		</Console>
	</Appenders>

	<Loggers>
		<!-- add this line to get debug output for this particular class -->
		<Logger name="dex.companyname.utility.Mymath" level="debug"/>
		<Logger name="dex.companyname.utility.CustomMath" level="debug"/>

		<Root level="info">
			<AppenderRef ref="STDOUT" />
			<AppenderRef ref="FILE" />
		</Root>
	</Loggers>

</Configuration>

runProgram.sh

#!/bin/bash

VMARGS="-Dlog4j.configurationFile=resources/log4j2-custom.xml "
CP=/tmp/version3/lib/log4j-api-2.6.2.jar:/tmp/version3/lib/log4j-core-2.6.2.jar:.
PGM=de.companyname.utility.CustomVersion

CMD="java -cp $CP $VMARGS $PGM"

echo $CMD
$CMD

References
http://logging.apache.org/log4j/2.x/manual/customloglevels.html

Note: The example was compiled with java 1.8 and uses log4j-api-2.6.2.jar and log4j-core-2.6.2.jar

Posted in programming | Tagged best practices, java, open source | Comments Off

Extreme measures

More Extreme measures

Manual jar creation

Running program

Simple build file

Dealing with the source code

software bundles

The solution

References

Bring in the pie’s

How LED cubes work

My own cubes logic

23017 IO Extender

This table is intended to give a brief overview. To have a full understanding you will need to reference the specification sheet for this chip.

Circuit diagram

Building the cube

layers

Assemble the layers

Drawbacks of this design

Alternatives

Code

picube.c

watchdog.sh

Makefile

Flashback to 2002

Back to the present

Quicksort

Java helps out

Converting over to threaded algorithm

Why does this work ?

Other important facts

Is this really worth the effort

Parting shot

Single threaded Quicksort

Threaded controller

Threaded sort

Bubble sort

Java sort

Generate sort data

Ant build script

The app I downloaded just doesn’t work. I am missing a lot of the photos that I took.

References

Secure PDF with password

Remove PDF password

Extract a single page

Extract all pages

Assembly of new PDF’s

Installing Windows software

PDFCrack Windows binary

PDFTK Windows binary

Installing Linux software

Install PDFCrack

Install PDFTK

Custom Levels

Create new level

Using new level

Examining the output

Other discoveries

Test program

References http://logging.apache.org/log4j/2.x/manual/customloglevels.html

Recent Posts

Categories

Archives

Meta

Tags

References
http://logging.apache.org/log4j/2.x/manual/customloglevels.html