Bash string splitting one-liner

I tried cmd challenge today and had fun writing Bash one-liners.

One particular challenge was new to me:

# The file split-me.txt contains a list of
# numbers separated by a ';' character.
# Split the numbers on the ';' character,
# one number per line.

I had to search for the solution and learned a new way to do it (reference here).

File containing delimited string:

# cat split-me.txt
# 1;2;3;4;5;6;7;8;9;10

My solution was this:

# IFS=';' read -ra array <<< $(cat split-me.txt);
   for i in "${array[@]}"; do echo $i; done

First we set IFS=’;’ so read command knows what delimiter it uses to create an array and we give it the file content (<<< and cat). Then we loop the array[@] (variable name given on first line) and print the numbers one by one to separate lines (echo $1 default action).

That cmd challenge is pretty fun but requires some experience in bash and patience to find all the required commands and parameters.

Awk whitespace trimming

I have to read an application version from the application config file using command line. The format is simple:

app.version = 739

Version info was read with following command:

# grep 'app.version' /path-to-file/file | awk '{print $3}'

This worked fine until the format changed a little bit:

app.version=new-and-better-version-string

These subtle whitespace changes cause the grep + awk logic to fail. The old simple implementation that relied on whitespace separated fields (“print 3rd field”) does not work anymore, we have to define the used delimeter to ‘=’ and trim possible whitespaces.

I had to change the awk part to trim the whitespaces (to make it work with both cases):

# grep 'app.version' /path-to-file/file | awk -F'=' '/=/{gsub(/ /, "", $2); print $2}'

Explanations on the awk gsub part in this stackoverflow thread.

This was a good opportunity to learn awk gsub feature. I had used awk + sed combo before but It’s good to know I can do this with awk alone.

Fix Windows 10 dual boot upgrade error

I have a laptop which has a dual boot setup with Windows and Linux. I had some problems updating the Windows 7 installation to Windows 10. My problem appeared in the form of a common C1900101-20017 error code in SAFE_OS phase.

The key step to get around the update error was restoring the Windows boot loader which had been replaced by Linux boot loader. I believe Windows 10 upgrade code assumes it has a full control on boot process and it falls back to the default (previously installed Windows version) whenever it can’t boot with the updated configuration. That’s pretty reasonable as it keeps the computer running instead of breaking the boot completely.

To get the upgrade to work we have to restore the Windows boot loader, update the system and then restore the old boot loader again.

Note that this applies to “Legacy BIOS” only, not UEFI.

Disclaimer: I am not responsible for any data loss and you should attempt to do this only if you have proper understanding on how the boot loaders work and how to restore them. If you haven’t done that before, there’s a good change you won’t be able to boot your computer for a while (until someone fixes it).

You should have proper backups made before attempting the update and some USB memory sticks. Do your homework on how to restore the Linux boot loader with your particular distribution. I am not going to give specific instructions for that.

Step 1: Create usb disks

Download the Boot-Repair-Disk and write it to USB stick. I also wrote Windows 10 upgrade USB in case I had to do some rescue stunts (fortunately I didn’t had to). You should also prepare your Linux distributions install USB as they often have restore functionalities included.

Step 2: Restore Windows boot loader

Reboot the computer with the Boot-Repair-Disk USB. Once the USB loads, use the repair tool to restore the mbr like instructed here. Note that this applies to “Legacy BIOS” only, not UEFI.

Once the mbr restore is done, you should be able to restart the computer and it should boot directly to Windows.

Step 3: Upgrade Windows

Now you should run the official Windows update procedure and hope for the best. If the boot phase goes well, the upgrade should go well.

Step 4: Restore Linux boot loader

The best option would be to use the same Linux distributions install USB and select the appropriate restore functionality. In my case I used the same Boot-Repair-Disk to restore Grub bootloader but there are many methods here that depends on your particular installation.

Hope this helps somebody.

Elm on Arch Linux

I wanted to try out Elm programming language on my Arch Linux laptop and here are the instructions for installing it.

The official Elm docs point to npm based install and that’s how I did it. First I had to install npm with normal pacman -S npm command.

When I had npm installed, Elm was quite easy to install with $ npm install -g elm. This requires root privileges as it installs the binaries to /usr/lib/ system directory.

After npm installs the binaries, close the terminal and open it again to get the elm command working. At this point I ran into error message telling me I didn’t have all the required libraries.

$ elm repl --help
/usr/lib/node_modules/elm/Elm-Platform/0.16.0/.cabal-sandbox/bin/elm-repl:
error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory


I did some googling and found this AUR package for the missing libtinfo . Quick install and then the elm binaries worked!

It is very clear that Elm is very young technology and its installation procedure is not yet matured. But hey, it’s working now! Now head to the Elm docs.

Time

How the perception of time affects workplaces? I catched myself pondering the question. Here are some thoughts on the subject.

Asap!!

The most usual time perception nowadays seems to be the “yesterday” attitude. Everything should be done already. Even if the thing is just a fresh unrefined and untested idea straight from the account managers mouth. These people often have only one type of priority markers in their posession, the highest one.

I have seen this kind of constant elevated urgency so often that I believe it only hurts the organisations that allow it to happen in their ranks. Of course there could be a real emergency but they are very rare. If the perceived emergency is constant, there’s something wrong in the organisation.

Mañana

The other end of the spectrum. Everything is done later. No things are afoot. This is bad of course because nothing is getting done. It may be a wonderful tactic for a vacation but it wont’t be good for a business.

If the priorisation is working this isn’t that big of an issue. On the other hand, if the priorisation does not work, important and useful things may end up to the graveyard.

The middle ground

The balance is found in the gray area, as it is often the case. People should be flexible with the time requirements and use the extreme cases rarely.

Time tracking

I have worked in organisations that tracked working hours in very different ways.

Some companies tracked every minute the workers spent inside the building. It worked in pure manufacturing work but even there it caused extra attention to the time spent at the premises. There were often queues at the clock and people counted seconds.

I have also worked in a place where there was no time tracking at all, the company just expected people to spend the days as agreed (37,5 hrs / week) but left the actual tracking and scheduling to the employees. I really liked it, unfortunately I have since changed venues.

Maximizing billable hours

Aka minimizing “unproductive” hours. Some business owners think that they will get the most profits from their investments by aiming for 100% billing rate. This is wrong and these people are sacrificing the long term success of their companies for the short term profit.

I think this attitude is a clear indication of the business owners lack of passion. Passion is good, lack of it isn’t. You might spend some time in a passionless company but it will always leave you cold and empty in the long run.

Brains need idle time

Have breaks, don’t fill your schedule to the max and leave time for “unproductive” tasks and your work will be more efficient and more pleasant (really important for your mental health). Endless pursue of better efficiency is for fools and slaves, you don’ want to be either.

Have a break now, you won’t regret it.

Command-line string manipulation tips

I just wanted to share some command line one-liners I used today to make my life easier.

Getting unique entries from a list

I had a long list of names and I had to find all the unique entries. I used common Linux commands,
sort and uniq. Input file is a text file with all the names on their own lines.

The file is first sorted by sort command and then uniq returns all the unique lines. The results are saved to output file.

# cat input-all.txt | sort | uniq > output-unique-entries.txt

Appending string to a list of entries

I had to convert a long list of dates in YYYY format to YYYY-MM-DD format with default dates of 1st Jan. I decided to do that with sed regex. Here is the command:

# cat input.txt | sed -e 's/^.*$/&-01-01/g' > output.txt

The command takes an input file (each date on its own line) and just appends ‘-01-01’ string after the year string. The & character is the key in the regex as is keeps the year in the result. The results are saved to output file.

Example: 1980 -> 1980-01-01

How to close telnet connections (Linux)

If you have done any telecom work, you have probably used telnet from command line. And then closed the whole terminal like everybody else because you have not figured out how to close the telnet session.

Telnet tells you what the escape character is but it does not help that much:

Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

Then how to quit the session? I finally found out today as I was reading this Elixir tutorial. It says:

My particular telnet client can be exited by typing ctrl + ],
typing quit, and pressing Enter, but your client may require
different steps.

And it works! Getting the required ctrl + ] required pressing ctrl + alt + 9 on my keyboard (Finnish mac kbd). Example here:

^]quit

telnet> quit
Connection closed.

Hope this helps you too if you use telnet.

Linux command line magic with find and xargs

Once again I used find command to find the files. This time I wanted to find all tar archives and extract them to current directory.

Had to do some googling on how to pass the filenames to tar but here it is. The magic requires using -I switch for xargs.

$ find . -name '*.tar.gz' | xargs -I{} tar xvfz {}

The xargs man page describes the -I switch as

 -I replace-str
              Replace occurrences of replace-str in the initial-arguments with
              names read from standard input.  Also, unquoted  blanks  do  not
              terminate  input  items;  instead  the  separator is the newline
              character.  Implies -x and -L 1.

To extract tar files I use the command tar xvfz <filename>. If your tar files are not gzip compressed, leave out the z option.

Video editing in Linux

I just finished a video editing project and I’d like to share some thoughts. If you want to use Linux to edit your videos, you are in trouble, there is no perfect solution. But don’t worry, it is still possible.

Most of the editors are awful

I used Pitivi, OpenShot and Kdenlive. I can’t really remember why I ditched Pitivi, perhaps it didn’t read my input files (dv video files). I used OpenShot for a while and managed to edit the first draft with it. But OpenShot is one of the buggiest programs I have ever used, it eventually crashed after every action I made to the timeline. I did some googling on the issue and it seems that the problem is quite wide-spread and the developers are not really doing their worth to fix them. Therefore I dumped OpenShot and moved forward.

After the other editors Kdenlive was a blessing. Finally a editor which acts how it should and didn’t crash all the time. Kdenlive is actually quite good editor and it should be your choice as well.

Rendering video is really slow

On my laptop my 3 hour video took 1-3 hours to render depending on the settings. I ended up rendering H.264/MPEG-4 video because I wanted to have wide playback support (Windows and Linux without additional codecs or software).

Ffmpeg is very versatile tool

I had to do some denoise work because some of my input material was shot in dark conditions. I tried to use Kdenlive effects for it but I ended up handling that task over to ffmpeg instead.

Sample ffmpeg command:

ffmpeg -i input.mp4 -filter:v “yadif,hqdn3d=4:4:3:3” -strict -2 output.mp4

Here we are using two filters, yadif and hqdn3d. Yadif is deinterlacing filter and I used it to handle my SD input material. Hqdn3d is the denoise filter and it takes a list of arguments. The arguments specify the level of different filtering algorithms, the values I have here are quite moderate and should provide reasonable picture quality. The quality is always compromised with filters and the actual values depend on the specific input material.

If you want to apply these filters to your project, I suggest creating several test clips with different values.

Conclusion

Video editing in Linux is possible but you have to select your tools properly.

Happy editing!

The key things for successful projects

This is my first post on management, a topic close to my heart.

Project management is not magic.

I believe the key things that make a successful project are (from the buyers point of view):

• Know what you want
• Respect other parties, expect the same from others
• Be open, share your vision
• Listen, be open to suggestions
• Realign your reality based on feedback

I have delivered few software projects on my career, some of them were more successful than the others. The projects were executed in different methodologies, some were more agile than others. The level of agility didn’t really mattered, it was more dependent on the client and the people.

From the client side the list would be:

• Know what they want to achieve with the project
• Respect them, they are the experts in their own business
• Listen them, don’t just try to silence them with your own sales pitch
• Share your suggestions and experience
• Build long-term relationships, don’t just rob them

In my opinion these are all acknowledged points in most of the companies but they are often neglected in real life situations. The pressure to optimize the profits often leads to aggressive selling of suboptimal solutions to customers needs and getting the maximum license and support costs.

On the other side the buyers are likely often looking for the most cost-optimized solution which would barely work for them. Again the pressure to optimize the short-term costs leads to half-assed projects which will cause hindered performance in the future.

I would like to believe we have now reached the bottom of this and eventually the company executives would start to drive their business by looking further than their next pay check. Their attitude is now on “I just work here” level together with a sense of grandiosity. To really do something valuable they need to change their attitude.

I don’t want to work for stuck-up people.

So to make your projects successful the company must first check its own priorities. If it’s only concerned in optimizing their owners cash flow the game is already lost and the projects will be painful. If the company does have other priorities their project success depends on the people who are in charge or the projects and their personal agendas. Use the bullet points in this post to check if your project management methods have the basics covered.

This post was written in software projects in mind but it probably correlates with other industries as well.

If you have any thoughts, please comment.