Using Lua scripts (Part 14): CLI commands timers and line editing

xiv: CLI commands timers and line editing

Part 1

This is a general piece of code for running a cli command and storing the output in a string.

local file = io.popen ("command")
output = file:read ("*a")
file:close ()

Say I wanted to get info about my partitions, I could run the df -h command like so.

local file = io.popen ("df -h")
output = file:read ("*a")
file:close ()

So if I were to do print (output), I would see in the terminal something like:

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              19G  4.9G   13G  28% /
udev                  3.9G  4.0K  3.9G   1% /dev
tmpfs                 1.6G  1.1M  1.6G   1% /run
none                  5.0M     0  5.0M   0% /run/lock
none                  4.0G  288K  4.0G   1% /run/shm
/dev/sda5             459G  223G  213G  52% /home
/dev/sr0              6.9G  6.9G     0 100% /media/Belk_4e_IRDVD

First problem is that the df -h command is being run every Conky cycle (just like if we were to use exec in the conky.conf). To get Lua to run the command in an execi type way, execute once per interval, we need to set up a timer.

SETTING A TIMER.

A Lua timer runs on the output of the Conky object ${updates}. In the conky.conf (conky.config = { ... };) you have the line update_interval = 1.

${updates} simply counts how may times Conky has updated. We can get this output in Lua through conky_parse:

updates = tonumber (conky_parse ("${updates}"))
interval = 10
timer = (updates % interval)

In the first line I'm using tonumber so that the output of the conky_parse is set as a number instead of a string.

The second line I have set the interval time. This is number of updates, so will only be equal to seconds if your update_interval in the conky.conf is set to 1.

The third line is the counting mechanism. What is going on here is that the number held in the string "updates" is being divided by the number held in the string "interval" and what is being output is the remainder.

So if updates = 6 and interval = 10, 6 / 10 = 0 and 6 / 10ths: timer = 6.

If updates = 38 and interval = 10, 38 / 10 = 3 and 8 / 10ths: timer = 8

Updates = 200, interval = 10: timer = 0.

Updates = 1737, interval = 60: timer = 57

And so on.

With interval = 10, timer will be the numbers 0 to 9 in repeating order. We can then use the output of timer in an if statement:

updates = tonumber (conky_parse ("${updates}"))
interval = 10
timer = (updates % interval)
if timer == 0 then
	-- do codeX
end

Timer will be 0 every 10 Conky cycles and at that time codeX will be run. So this is analogous to ${execi 10 code} in the conky.conf.

BUT one problem still remains. Unlike ${execi 10}, the code in our if statement will not automatically run on Conky start (or Conky restart after saving the conky.conf). To get around this we have to think about how the Lua script is being run by the conky.conf.

A simple example Lua script

require("cairo")
require("cairo_xlib")

print("This is outside of the function")

function conky_main()
	if conky_window == nil then
		return
	end

	local cs = cairo_xlib_surface_create(
		conky_window.display,
		conky_window.drawable,
		conky_window.visual,
		conky_window.width,
		conky_window.height
	)

	cr = cairo_create(cs)
	--
	print("This is inside the function")
	--
	cairo_destroy(cr)
	cairo_surface_destroy(cs)
	cr = nil
end

Run that code in the Conky.conf like this:

conky.config = {
	-- Conky settings above.
	lua_load = '/path/file.lua',
	lua_draw_hook_pre = 'main',
};

conky.text = [[ ]];

Watch the terminal as you launch Conky with conky -c and you will see the following:

mcdowall@mcdowall-desktop:~/Desktop$ conky -c ~/lua/conky_test
This is outside of the function
conky: desktop window (e00022) is subwindow of root window (15d)
conky: window type - normal
conky: drawing to created window (0x2e00001)
conky: drawing to double buffer
This is inside the function
This is inside the function
This is inside the function
This is inside the function
This is inside the function
This is inside the function
This is inside the function
This is inside the function

When Conky starts or restarts after a save, the entire Lua script is read from top to bottom, so on Conky start (and restart) we see "This is outside of the function", but ONLY on Conky start.

Once Conky has started, and on every Conky update following, only the code contained in the function conky_main is run so we see "This is inside the function" repeated every Conky cycle.

We can use this to our advantage to activate Lua code on Conky start

NOTE: this feature can be used for lots of other things too.

require("cairo")
require("cairo_xlib")

conky_start = 1

function conky_main()
	if conky_window == nil then
		return
	end

	local cs = cairo_xlib_surface_create(
		conky_window.display,
		conky_window.drawable,
		conky_window.visual,
		conky_window.width,
		conky_window.height
	)

	cr = cairo_create(cs)

	updates = tonumber(conky_parse("${updates}"))
	interval = 10
	timer = (updates % interval)
	if timer == 0 or conky_start == 1 then
		print("update number= " .. updates)
		print("You will see this at Conky start and then at 10 cycle intervals")
		conky_start = nil
	end

	cairo_destroy(cr)
	cairo_surface_destroy(cs)
	cr = nil
end

Now you see this in the terminal:

mcdowall@mcdowall-desktop:~/Desktop$ conky -c ~/lua/conky_test
conky: desktop window (e00022) is subwindow of root window (15d)
conky: window type - normal
conky: drawing to created window (0x2e00001)
conky: drawing to double buffer
update number= 1
You will see this at Conky start and then at 10 cycle intervals
update number= 10
You will see this at Conky start and then at 10 cycle intervals
update number= 20
You will see this at Conky start and then at 10 cycle intervals
update number= 30
You will see this at Conky start and then at 10 cycle intervals

NOTE the line conky_start = nil, as it is important.

Even though the line conky_start = 1 only runs at startup, the string conky_start is retained within the Lua script. If we were to put print (conky_start) into our function (above where we set it nil), we would get "1" printed in the terminal. So unless we change the value of conky_start, it will retain the value of 1 and the if evaluation (if timer == 0 or conky_start == 1 then) will be passed every cycle and the code will be run every cycle.

You could set conky_start to anything other than 1, but setting it to nil has the added benefit of deleting the variable. Fewer variables floating around the better.

NOW that we can control our commands, lets get back to running them!

We ran the df -h command and stored the output in a string BUT we want to make the information in that string usable.

For example display the mount point for each item using cairo_show_text. Or we want to take the use% for each entry and display that information as a bar or other indicator. To do this we need to extract the various bits and pieces out of the string and into other strings or tables.

One of the best methods for doing this in Lua is the string.find command. For example, to get the mount points for each partition listed from df -h:

local file = io.popen ("df -h")
output = file:read ("*a")
file:close ()
start, finish, mount1 = string.find (output,"%%%s*(/[%d%a%p]*)\n")
start, finish, mount2 = string.find (output,"%%%s*(/[%d%a%p]*)\n",finish)
start, finish, mount3 = string.find (output,"%%%s*(/[%d%a%p]*)\n",finish)
print (mount1)
print (mount2)
print (mount3)

In the terminal, you will see:

/
/dev
/run

These are the first 3 mount points in the list given by df -h.

THE HARD PART is what to put into the string.find command and when writing code there are usually several ways to get the same result.

There are plenty of resources to learn about string.find and other string manipulation commands: Tutorial, The 5.1 manual Lua Manual Sting Manipulation and Lua Manual Patterns. The Lua users wiki Lua Manual String Lib Tutorial.

The way that I use string.find, 3 things are returned from the command in this order.

• The position in the string where that match started.
• The position in the string where the match ended.
• The captured part of the string (the part that contains the info we want).

This is why I have started my lines: start, finish, mount =, then the command itself: string.find.

And then we send the string.find command all the bits and pieces it needs inside round brackets () The first thing to send it is the string that contains the information we want to get out, which in this case is called "output".

The next part is where the matching happens.

LETS look at the results from "df -h"

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              19G  4.9G   13G  28% /
udev                  3.9G  4.0K  3.9G   1% /dev
tmpfs                 1.6G  1.1M  1.6G   1% /run
none                  5.0M     0  5.0M   0% /run/lock
none                  4.0G  812K  4.0G   1% /run/shm
/dev/sda5             459G  223G  213G  52% /home
/dev/sr0              6.9G  6.9G     0 100% /media/Belk_4e_IRDVD

We need to identity a feature of the output that we can use in a match. Ideally we need a feature to match before the data we want and a feature to match following the data.

In this case preceding the mount point on each line is the "Use%" number. This is the only place in a line where you see the % symbol, and for every line in the output the mount point for the partition is preceded by a % and a space.

The mount point itself is the last thing in the line. In this case a newline character is \n, so this gives us something to match after the mount point.

NOW we have to look at pattern matching. The standard patterns you can search for are:

 . --- (A dot) Represents all characters.
%a --- Represents all letters.
%c --- Represents all control characters.
%d --- Represents all digits.
%l --- Represents all lowercase letters.
%p --- Represents all punctuation characters.
%s --- Represents all space characters.
%u --- Represents all uppercase letters.
%w --- Represents all alphanumeric characters.
%x --- Represents all hexadecimal digits.
%z --- Represents the character with representation 0.

Important: the uppercase versions of the above represent the complement of the class, eg %U represents everything except uppercase letters, %D represents everything except digits.

There are some "magic characters" (such as %) that have special meanings. These are: ^ $ ( ) % . [ ] * + - ?

If you want to use those in a pattern (as themselves) you must precede them by a % symbol, eg. %% would match a single %.

You can build your own pattern classes by using square brackets, eg:

[abc] ---> Matches a, b or c.
[a-z] ---> Matches lowercase letters (same as %l).
[^abc] ---> Matches anything except a, b or c.
[%a%d] ---> Matches all letters and digits.
[%a%d_] ---> Matches all letters, digits and underscore.
[%[%]] ---> Matches square brackets (had to escape them with %).

The repetition characters are:

+  ---> 1 or more repetitions (greedy).
*  ---> 0 or more repetitions (greedy).
-  ---> 0 or more repetitions (non greedy).
?  ---> 0 or 1 repetition only.

The standard "anchor" characters apply:

^  ---> Anchor to start of subject string.
$  ---> Anchor to end of subject string.

SO lets look at the pattern matching I used: "%%%s*(/[%d%a%p]*)\n".

My data is preceded by % and that is what I am matching with %%%s.

%% is a % character (remember that % itself is a Lua "magic" character and needs to be escaped with a preceding %).

%s is a space character.

CAPTURES. Captures are specified using round brackets (). Looking at my output I see that all of my mount points start with a back-slash /. So I put this as the first character in my capture brackets. One reason for this is that I DON'T want to capture the "Mounted on" from the first line. Since there is no / in that line, it shouldn't match.

After the / there are many different things that might appear in the mount point of a partition. In particular you might get any combination of Numbers, letters and punctuation, which are represented here with [%d%a%p]*, ie any combination of numbers, letters or punctuation repeated 0 or more times.

You shouldn't get any spaces. If you do have spaces in the names of partitions then you would need to add %s into the brackets.

So my capture is as follows: (/[%d%a%p]*).

Following the capture is /n because I want to stop the string.find command when it reaches the end of the line and matches a newline character.

I will continue with explaining the string.find lines I used as well as other ideas on how to do the data extraction and methods of "automating" the process at a later date.

Part2

Here is the output of "df -h" as stored in the string output

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              19G  4.9G   13G  28% /
udev                  3.9G  4.0K  3.9G   1% /dev
tmpfs                 1.6G  1.1M  1.6G   1% /run
none                  5.0M     0  5.0M   0% /run/lock
none                  4.0G  288K  4.0G   1% /run/shm
/dev/sda5             459G  223G  213G  52% /home
/dev/sr0              6.9G  6.9G     0 100% /media/Belk_4e_IRDVD

My first data extraction line:

start, finish, mount1 = string.find (output,"%%%s*(/[%d%a%p]*)\n")
print (mount1) --> /
print (finish) --> 101

string.find finished matching my expressions at position 101.

My second data extraction line:

start, finish, mount2 = string.find (output,"%%%s*(/[%d%a%p]*)\n",finish)
print (mount2) --> /dev

Using the value of "finish" from the previous match at the end of this string.find command tells string.find to START searching at that position onward. The default, if not start position is given, is that string.find begins at position 1. The result is that the second mount point is returned.

This could be repeated so on and so forth until you have all the mountpoints from df -h.

BUT what if you didn't know how many lines you were dealing with? This will most likely be the case as partitions may be mounted or unmounted.

READING THE OUTPUT LINE BY LINE. This can be an extremely useful technique. This is the general form for the code:

local file = io.popen ("command")
    for line in file:lines () do
        -- Code
    end
file:close ()

What this does is to take each line from the output of the command and apply the code to it individually. For our df -h command we could do this:

local file = io.popen ("df -h")
    for line in file:lines () do
		print ("This is a line ".. line) -- line is the name of the string that
										-- 		contains each line from the
										-- 		command "df -h"
    end
file:close ()

And get this in the terminal:

This is a line Filesystem            Size  Used Avail Use% Mounted on
This is a line /dev/sda1              19G  4.9G   13G  28% /
This is a line udev                  3.9G  4.0K  3.9G   1% /dev
this is a line tmpfs                 1.6G  1.1M  1.6G   1% /run
This is a line none                  5.0M     0  5.0M   0% /run/lock
This is a line none                  4.0G  640K  4.0G   1% /run/shm
This is a line /dev/sda5             459G  223G  213G  52% /home
This is a line /dev/sr0              6.9G  6.9G     0 100% /media/Belk_4e_IRDVD

Looking at the print command (print ("this is a line "..line)):

I have used .. to join together some text ("This is a line") with a string called line. The string named line holds each line from the output of the df -h command, one after another, in order. This is a different kind of for loop. The loop runs until it has read every line and applied any code within the loop then the loop breaks and the rest of the code continues.

In the loop I simply had the text "This is a line" added onto the front of each line, hence the output.

Let's extract the "Used" value from each line.

For this method I am going to extract the values into a table and I am going to put the table.

updates = tonumber (conky_parse ("${updates}"))
interval = 10
timer = (updates % interval)
if timer == 0 or conky_start == 1 then
used_table = {}
local file = io.popen ("df -h")
for line in file:lines () do
	s, f, used = string.find (line,"[%d%p]*%u%s*([%d%p%a]*)%s")
	table.insert (used_table, used)
end
file:close ()
conky_start = nil
end
print (used_table[2]) --> 4.9G

This is the table I set up to receive the values used_table = {}. It is important that the table is INSIDE the timer-controlled section because whenever the script runs the line: used_table = {}, any contents in the table. So if the table was outside of the timer, it would be blanked every cycle. BUT data will only be written to the table once every 10 cycles (interval = 10). So 9 cycles out of 10 the table will be blank, which is no good.

Here is the string.find line: s, f, used = string.find (line,"[%d%p]*%u%s*([%d%p%a]*)%s").

I have shortened the names of the start and finish receiving strings to s and f.

NOTE: if the results of the start and finish position of the match are of no interest string.match could be used as follows used = string.match (line,"[%d%p]*%u%s*([%d%p%a]*)%s").

string.match only returns the captured part of the string.

BUT I'm going to stick with string.find.

The first thing we feed the string.find command is the string that holds the information we want to look at: line.

Then looking at the output of df -h, we need to identify the before and after features to match

/dev/sda1              19G  4.9G   13G  28% /

BEFORE the "Used" value is the "Size" value. The Size value looks to be a combination of numbers and punctuation (decimal point) followed by an upper case letter denoting the unit followed by one or more spaces.

So that is what I put into my matching statement before my capture brackets: [%d%p]*%u%s*

• [%d%p]* = any combination of 0 or more numbers and punctuation.
• %u = a capital letter.
• %s* = 0 or more spaces.

AFTER the "Used" value there are always at least one space, and that is what I will use to end the match after the capture brackets: %s.

The capture brackets themselves: ([%d%p%a]*) any combination of 0 or more numbers, punctuation and letters.

The loop cycles through the lines, extracts the Used value and then: table.insert (used_table, used) puts the values into the table in order.

In the above case the first entry in the table is actually "filesystem" which, by applying the search criteria to the very first line ...

Filesystem            Size  Used Avail Use% Mounted on

... should be self explanatory.

You could look at what was in the table like this:

table_length = tonumber (#used_table)
for i = 1, table_length do
	print (table[i])
end

NOTE: if we only wanted to get the number part of the Used value, there is a catch in this case: if the Used value is greater than 0 the value always ends in an uppercase letter BUT when Used = 0 as in the line ...

none                  5.0M     0  5.0M   0% /run/lock

... we only have the 0, and no letter. This, however would work to get just the number: s, f, used = string.find (line,"[%d%p]*%u%s*([%d%p]*)%u*%s").

However, the expression within the capture brackets no longer matches the first line from df -h, so print (used_table[1]) prints 4.9 (the value for the / partition)`.

Getting Data from webpages

The same methods as used to get the data from the output of df -h can be used to extract information from a web page. The command to use here would be:

local file = io.popen ("curl 'web address'")
output = file:read ("*a")
file:close ()

NOTE: be careful with your quotation marks. The curl command requires that you enclose the web address you want to access within quotes, but the Lua code also requires that the entire command you want to run be enclosed in quotes. Both types of quotes (double " and single ') can be used to get around this requirement and it doesn't matter which way around you use them -- eg enclose the command in one type and the web address in another.

If you use a text editor like gedit, once you save your file as a .lua, you will see that gedit uses different colors to highlight different parts of the text For me, quoted text in Lua shows pink in Gedit (Gedit = pink, Geany = yellow).

There are several ways you could do this. You can set the web address as a string (for example if you wanted to construct a settings section separate to the main code). This can help you get round any problems related to quotes, also.

updates = tonumber (conky_parse ("${updates}"))
interval = 100
timer = (updates % interval)
if timer == 0 or conky_start == 1 then
	used_table = {}
	web = "http://crunchbanglinux.org/forums/search/recent/"
	file = io.popen ("curl "..web)
	output = file:read ("*a")
	file:close ()
	conky_start = nil
	print (output)
end

What gets printed in the terminal is the contents of "output", which is the entire source code for the page.

NOW we can start working on getting the data out. Say we want to get the topmost thread title from the "Active Topics" page on the Crunchbang forum: Crunchbang topic list.

The first thing is to open the page, once as you would view it regularly and once after right clicking somewhere on the page and selecting "View Page Source".

Right now, the top thread in the list is titled: Indefinite Hang on Reboot/Shutdown.

Go to the source code and search for those exact terms, then look around a bit at the code.

    </div>
    <div class="main-content main-forum forum-noview">
        <div class="main-item odd main-first-item new">
            <span class="icon new"><!-- --></span>
            <div class="item-subject">
                <h3 class="hn"><span class="item-num">1</span> <a href="http://crunchbanglinux.org/forums/topic/21773/indefinite-hang-on-rebootshutdown/">Indefinite Hang on Reboot/Shutdown</a> <span class="item-nav">( <em class="item-newposts"><a href="http://crunchbanglinux.org/forums/topic/21773/indefinite-hang-on-rebootshutdown/new/posts/" title="Go to the first new post since your last visit.">New posts</a></em> )</span></h3>
                <span class="item-starter">by <cite>drewdle</cite></span>
            </div>

Below that you will see the other topics in the list. The first thing we need to look for is something unique that identifies this topic as the first one ... I see: class="item-num">1</span>.

Before the info I want, and when I search for that expression I find that it is unique. The only downside is the quotes, which might make writing a matching expression more difficult. Ignoring the quoted part: >1</span> is also unique, so we will use that.

ALSO, directly before the info I want there is this: /">. This appears before the title of every entry in the list and can help us to get only the title out from the code.

And directly after the info I want I see: </a> <span class=. This appears to be a constant, so should give us a predictable place to stop our search.

THE FIRST THING to do is to replace any actual spaces in our search terms with %s, so </a> <span class= becomes </a>%s<span%sclass=, and replace any other troublesome characters: /"> becomes /%p> (%p is a punctuation character).

Another thing to look out for is if there are any Lua magic characters that need to be escaped with %.

lets put together our search pattern:

BEFORE the capture brackets we have: >1</span> followed by any combination of 0 or more numbers, letters, punctuation and spaces until we get to /"> which becomes: >1</span>[%d%a%p%s]*/%p>.

AFTER the capture brackets we have: </a> <span class= which becomes: </a>%s<span%sclass=.

INSIDE the capture brackets we are going to match any combination of 0 or more numbers, letters, punctuation and spaces: ([%d%a%p%s]*).

The entire code to get the first entry from the "Active topics" page on the Crunchbang forum is then:

updates = tonumber (conky_parse ("${updates}"))
interval = 100
timer = (updates % interval)
if timer == 0 or conky_start == 1 then
	used_table = {}
	local web = "http://crunchbanglinux.org/forums/search/recent/"
	local file = io.popen ("curl "..web)
	output = file:read ("*a")
	file:close ()
	local s, f, topic1 = string.find (output,">1</span>[%d%a%p%s]*/%p>([%d%a%p%s]*)</a>%s<span%sclass=")
	conky_start = nil
	print (topic1)
end

Which outputs in the terminal:

mcdowall@mcdowall-desktop:~/Desktop$ conky -c ~/lua/conky_test
Conky: desktop window (e0aafa) is subwindow of root window (15d)
Conky: window type - normal
Conky: drawing to created window (0x2a00001)
Conky: drawing to double buffer
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 38635    0 38635    0     0  99067      0 --:--:-- --:--:-- --:--:--  118k
Indefinite Hang on Reboot/Shutdown

These lines ...

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 38635    0 38635    0     0  99067      0 --:--:-- --:--:-- --:--:--  118k

... show curl working.