org Archive

Tables in Org-mode

Here’s another nice thing about blogging in org-mode: easy tables. You can create a table in org-mode as simple as creating the first line with pipes between each item, and then tab for the next cell or line. Formulas are also pretty simple. Then exporting to a blog turns that into a nice HTML table for you.

For instance, here’s the table I’ve been using to track my garden harvest so far this year, as it looks in org-mode as of today:

| date             | harvest      | grams | pounds | price | total $ |
|------------------+--------------+-------+--------+-------+---------|
| [2018-04-24 Tue] | asparagus    |   572 |   1.26 |  3.00 |    3.78 |
| [2018-04-26 Thu] | asparagus    |   556 |   1.22 |  3.00 |    3.66 |
| [2018-04-28 Sat] | asparagus    |   663 |   1.46 |  3.00 |    4.38 |
| [2018-04-30 Mon] | asparagus    |   508 |   1.12 |  3.00 |    3.36 |
| [2018-05-02 Wed] | asparagus    |   488 |   1.07 |  3.00 |    3.21 |
| [2018-05-05 Sat] | asparagus    |   460 |   1.01 |  3.00 |    3.03 |
| [2018-05-08 Tue] | asparagus    |   684 |   1.50 |  3.00 |    4.50 |
| [2018-05-08 Tue] | green onions |   128 |   0.28 |  2.00 |    0.56 |
| [2018-05-08 Tue] | marshmallow  |   100 |   0.22 |  1.00 |    0.22 |
| [2018-05-12 Sat] | asparagus    |  1066 |   2.34 |  3.00 |    7.02 |
| [2018-05-12 Sat] | lettuce      |   150 |   0.33 |  1.00 |    0.33 |
| [2018-05-15 Tue] | asparagus    |   736 |   1.62 |  3.00 |    4.86 |
| [2018-05-20 Tue] | asparagus    |   686 |   1.51 |  3.00 |    4.53 |
| [2018-05-20 Tue] | lettuce      |   150 |   0.33 |  1.00 |    0.33 |
| [2018-05-22 Tue] | asparagus    |   251 |   0.55 |  3.00 |    1.65 |
|------------------+--------------+-------+--------+-------+---------|
| Totals           |              |  7198 |  15.82 |  2.53 |   40.02 |
#+TBLFM: $4=$3/455;%.2f::$6=$4*$5;%.2f::$5=$5;%.2f:: @19$3=vsum(@4..@-1)::@19$5=vmean(@4..@-1);%.2f

The cryptic stuff at the bottom is a list of formulas to do things like convert grams to pounds and calculate and total the prices, so it stays up to date whenever I add lines.

And here’s how it looks in the web page as exported through ox-hugo, with no tinkering except to set a few general CSS table styles:

date harvest grams pounds price total $
[2018-04-24 Tue] asparagus 572 1.26 3.00 3.78
[2018-04-26 Thu] asparagus 556 1.22 3.00 3.66
[2018-04-28 Sat] asparagus 663 1.46 3.00 4.38
[2018-04-30 Mon] asparagus 508 1.12 3.00 3.36
[2018-05-02 Wed] asparagus 488 1.07 3.00 3.21
[2018-05-05 Sat] asparagus 460 1.01 3.00 3.03
[2018-05-08 Tue] asparagus 684 1.50 3.00 4.50
[2018-05-08 Tue] green onions 128 0.28 2.00 0.56
[2018-05-08 Tue] marshmallow 100 0.22 1.00 0.22
[2018-05-12 Sat] asparagus 1066 2.34 3.00 7.02
[2018-05-12 Sat] lettuce 150 0.33 1.00 0.33
[2018-05-15 Tue] asparagus 736 1.62 3.00 4.86
[2018-05-20 Sun] asparagus 686 1.51 3.00 4.53
[2018-05-20 Sun] lettuce 150 0.33 1.00 0.33
[2018-05-22 Tue] asparagus 251 0.55 3.00 1.65
Totals 7198 15.82 2.53 40.02
  • price notes
    • asparagus price from Mill Creek Farm in season
    • green onions guessed from online
    • no idea on marshmallow, so plugged in $1
    • have to look up lettuce price too

Change to org-agenda-time-grid in Org 9.1

Another small one that others might be searching for. The upgrade to Org 9.1 included a change to the arguments in org-agenda-time-grid, adding a new one and rearranging them a bit. This was my previous setting (from Bernt Hansen’s config):

(setq org-agenda-time-grid (quote ((daily today remove-match)
                                   #("----------------" 0 16 (org-heading t))
                                   (0900 1100 1300 1500 1700))))

And now it’s this:

(setq org-agenda-time-grid (quote ((daily today remove-match)
                                   (0900 1100 1300 1500 1700)
                                   "      " "................")))

It doesn’t seem necessary to insert org-heading into the last string anymore; it’s doing that on its own, even though the docstring for the variable doesn’t mention that. I changed those two strings just for style purposes. The first, shorter string comes after the time in the scheduler, and spaces look better there.

Switched from ido-mode to ivy-mode for org-mode completion

I used ido-mode for completion in org-mode for a long time, based on settings I got from Bernt Hansen’s Org Mode config. Recent changes to Org for version 9 have broken a few things. One is that org used to have its own hook into ido-mode for completion on things like refile tasks, using the variable org-completion-use-ido. That no longer exists. The docs say it can use a completion engine via completing-read, but while researching how to do that, I ran across ivy-mode. It’s similar to ido, but nicer in some ways. To enable it, just remove the ido stuff in your emacs/org-mode config, and add this somewhere in your .emacs:

(ivy-mode 1)

That seems to enable it everywhere useful, so far. I like the way it displays choices better than ido, so I’m already getting comfortable with it after a couple days.

Fixing Org-protocol issue with conkeror

I have a function in conkeror that saves a web page’s URL and title, along with any selected text at the time, in Emacs/Org-mode as a captured task, when I hit C-c r. It does this by feeding an org-protocol command through emacsclient. A recent upgrade of org-mode broke it, so I had to change it up a bit. The function in my .conkerorrc used to look like this:

function org_capture (url, title, selection, window) {
    var cmd_str = 'emacsclient "org-protocol://capture:/w/'+
        url + '/' + title + '/' + selection + '"';
    if (window != null) {
        window.minibuffer.message('Issuing ' + cmd_str);
    }
    shell_command_blind(cmd_str);
}

And now it looks like this (plus the interactive function and key definition to call it, which hasn’t changed):

function org_capture (url, title, selection, window) {
    var cmd_str = 'emacsclient "org-protocol://capture?template=w&url='
                  + url + '&title=' + title + '&body=' + selection + '"';
    if (window != null) {
        window.minibuffer.message('Issued: ' + cmd_str);
    }
    shell_command_blind(cmd_str);
}
interactive("org-capture",
            "Clip url, title, and selection to capture via org-protocol",
    function (I) {
        org_capture(encodeURIComponent(I.buffer.display_uri_string),
                    encodeURIComponent(I.buffer.document.title),
                    encodeURIComponent(I.buffer.top_frame.getSelection()),
                    I.window);
    });
define_key(content_buffer_normal_keymap, "C-c r", "org-capture");

Basically, they went from a format that separated the items by slashes to one with key=value pairs, like URL format. I also had to make a change in my emacs config to org-capture-templates (just showing the one template here), from this:

("w" "org-protocol" entry (file "~/work/org/refile.org")
 "* TODO Review %c :CAP:\n%U\n\n%i" :immediate-finish t)

To this:

("w" "org-protocol" entry (file "~/work/org/refile.org")
 "* TODO Review %:annotation :CAP:\n%U\n\n%i" :immediate-finish t)

Not much change there; just seems like they changed the %c template marker to %:annotation. Works perfectly again now.

Org-Mode: Return to Task Buffer When Closing Email

This is a small thing, but it’s been bugging me for a while, so I’m glad I finally took the time to find a solution.

When I read my email, I don’t respond to anything on the spot. Every message that requires a reply or any other action gets refiled as an org-mode task with a header, timestamp, and link to the message. When I’m finished going through mail and refiling everything that needs an action, I then go to my org-mode agenda, which shows those tasks, and clock each one in while I handle it. That way, the time I spend on each email is recorded, and I don’t get bogged down in the middle of checking email without knowing what else I have to do. If one of them needs priority over the others, I can tackle it first without worrying that unknown emergencies are waiting further down in my email.

Each of these tasks is saved with an underlined link directly to the message in Gnus, as in the following example. That’s very handy, since I can clock-in the task and then one keystroke gets me right back into the email, where I can reply or do whatever I need to do.

* NEXT Respond to Castalia House <books@castaliahouse.com> on NEW RELEASE: APPENDIX N: A Literary History of D&D
SCHEDULED: <2017-01-17 Tue>
[2017-01-17 Tue 10:01]
<u>Email from Castalia House: NEW RELEASE: APPENDIX N: A Lit</u>

The annoyance comes when exiting the email. Hitting ‘q’ to leave the summary buffer takes me back to the Gnus Group buffer, not back to the buffer containing the org-mode task I’m working on. In almost all cases, I want to return to that task to make notes and/or mark it done. I don’t need to go to the Group buffer, because I’m not in “email reading mode” now.

At first I thought I could add something to one of the hooks that runs when exiting a Gnus group. But both of the hooks, gnus-summary-exit-hook and gnus-group-exit-hook, run in gnus-summary-exit before the buffer is switched back to the Group buffer. No hooks run after that, and there’s no option to have it not switch to that buffer. So I would have to hack the function itself, which is a poor solution.

Then I remembered the “advice” ability of Emacs, to define a function to run after any other function, whether the first function has hooks for it or not. This worked nicely. A single line did the trick:

(defadvice gnus-summary-exit (after gnus-summary-exit-after activate) (bury-buffer))

What this does is to “bury” the Group buffer after gnus-summary-exit has returned to it. Burying a buffer just moves it to the bottom of the buffer list, which leaves the next buffer on top – in this case, the buffer I was in before I went into the email, which is the buffer containing the org-mode task. My cursor is even right where I left it in the task, ready to finish the action.

The only downside to this solution is that if I’m reading news in Gnus, each time I leave a group, it’s going to bury the Group buffer, and I’ll have to select it again. I almost never do that, though, so that’s not an issue for me.

Saving Blog Comments in Org-Mode with Clocking

After using </emacs/org-mode/saving-blog-comments-in-org|my mix of edit-server and org-capture for blog comments> for a while (see that page for instructions on setting up the org-mode template), I realized I’d like to have org-mode clock the time I spend commenting on blogs. I already have a ‘Reading Blogs’ task that I clock in for that, but it’d be nice to clock commenting separately, since that’s creating content. Plus, org-mode’s clocking is so nice that I like to use it as well as possible.

At first I wasn’t sure how to do it, though. Org-capture has clocking built-in, but I couldn’t use that, because I’m not editing in the org-capture buffer. Editing the comment is done in an edit-server buffer, then passed to org-capture. So clocking in the org-capture buffer would only count the split-second it takes to file the task.

I dug through the source of both edit-server.el and org-capture.el, looking for a way to tie the two together. I thought I might be able to get edit-server.el to do the clocking itself. Then I thought maybe edit-server.el could pass the text on to be edited in the org-capture buffer, which could then send it back to edit-server to be sent back to the browser. I think either of those would be doable, but they’d require hacking on the source of one or both modes, which I wanted to avoid if possible.

Then I had a new thought: I don’t actually need to clock the individual comments. In fact, I’d rather not break it down that fine; I just want to keep track of time spent on blog commenting in general. That gave me the answer: create a new task called ‘Blog Commenting’, and clock that task in while I’m working in the edit-server buffer, and clock out that task when I exit it. That turned out to be fairly easy to do.

First, I needed the ‘Blog Commenting’ task, and a way to clock it in from anywhere else in Emacs. I created it as a task in comments.org, above the datetree section, and gave it a unique ID property with org-id-get-create (creates an ID for the current task if one doesn’t already exist). That gave me this (your ID value will be different):

* NEXT Blog Commenting
:PROPERTIES:
:ID:       0312e0bf-6c55-4657-b2aa-5d92817d4d3f
:END:

Now I need to add a function to the hook that edit-server runs when starting an edit buffer on a new comment. This code in my .emacs does that. org-id-find returns a marker to the task containing the ID I just created in the Blog Commenting task (and copy-pasted to this code), and org-with-point-at runs org-clock-in on that task:

(defun ajb/clock-in-blog-commenting-task ()
  "Clock in the special commenting task while editing a blog comment
via edit-server."
  (org-with-point-at (org-id-find "0312e0bf-6c55-4657-b2aa-5d92817d4d3f" 'marker)
    (org-clock-in)))
(add-hook 'edit-server-start-hook 'ajb/clock-in-blog-commenting-task)

Now to clock-out the task when I close the buffer. I just needed to add two lines to the function I’d already written, but here’s the whole thing for simplicity’s sake, again from my .emacs:

(defun ajb/save-edit-server-buffer ()
  "Save the edit-server buffer as an org-mode entry via org-capture
   when it is saved with C-c C-c.
   Should be called from edit-server-done-hook."
  (when (not (string-match "edit-server" (buffer-name)))
    (goto-char (point-min))
    (insert (buffer-name) "\n")
    (mark-whole-buffer)
    (org-capture nil "e")
    (goto-char (point-min))
    (kill-line 1)
    (if (equal org-clock-current-task "Blog Commenting")
        (org-clock-out))))
(add-hook 'edit-server-done-hook 'ajb/save-edit-server-buffer)

The new part is the if statement near the end. I have it check to make sure the current task is the Blog Commenting task before clocking out of it. That way, if I’ve already closed the task, or if I’ve switched to some other task in the middle of writing a comment, I won’t be clocking-out of some other task without meaning to.

It works just like it did before, except now I have a running total of how much time I spend writing blog comments, which can be included in my daily/weekly/monthly time reports and other stats. I look forward to seeing how much time I spend writing comments, compared to reading blogs and other work-related activities.

If you have questions or need help getting this working, please contact me.

Org-Mode: Saving Blog Comments

Update: I added clocking as a feature later.

I like to save my blog comments and other things I post to the web, for a couple reasons. First, once in a while a comment form fails, especially at sites like Blogspot. You spend 20 minutes writing and proofreading a comment, press the submit button, and poof – an error page and your comment is lost. So saving them provides a backup. But I also like to keep them in case I want to refer back to something I wrote later. Back in the days of Usenet and mailing lists, it was just standard to save outgoing copies of everything. But on the web, it’s rare for a site to have that kind of functionality, and it can be difficult to track a comment down with a web search if you can’t remember the exact wording – and there’s always the chance a site will go away.

So for a few years now, I’ve been using an extension for Chromium called <https://chrome.google.com/webstore/detail/comment-save/ndmcbhmmonjkclhmeidccodfhlifmmco|Comment Save>. It’s supposed to save everything you submit in a textarea (the standard multi-line text field). There are some problems with it, though. It’s flaky – sometimes when I go looking for a comment, it doesn’t have it. It also doesn’t seem to work with some of the sites where I post frequently. It’s also only for Chromium/Chrome, and I’d like to shift to Qupzilla for most of my browsing.

I didn’t see any promising alternatives, so I started thinking about other ways to do it. I use another extension called <http://www.emacswiki.org/emacs/Edit_with_Emacs|Edit with Emacs>, which allows you to use Emacs to edit textarea fields. It puts a button next to each textarea, and when you press it, the textarea’s contents pop up in an emacs buffer (via a small daemon called edit-server). You edit it there, save it with C-c C-c, and the edited text appears back in the textarea in Chromium. It works pretty nicely, and editing in Emacs is much nicer than editing in a standard text box, but I haven’t used it much up to now.

So I started thinking: when I hit C-c C-c to tell edit-server to send the new text back to Chromium, surely I could hack into that code to have it save the buffer to a file. Sure enough, edit-server even provides a hook, edit-server-done-hook, which runs at that point. So I started thinking about how I would arrange the saved comments into a file tree, perhaps arranged by URL/year/month/day, or something like that. Then it hit me – why not save them into org-mode? It already has journal-style capture functionality, with the ability to save entries by date. Then instead of having them scattered through a bunch of files, I’ll have them all arranged in org-mode where I can use familiar tools to search them, back them up, and do whatever else might come to mind.

This sounded very promising, so I got to work. First, I needed a capture template for capturing these entries into a date-arranged tree. I added this to the list in org-capture-templates:

("e" "Edit with Emacs" entry (file+datetree "~/work/org/comments.org")
  "* %U\n%i\n\n" :immediate-finish t)

Here’s what the elements in that template do:

“e” The key that org-capture will use to specify this template.
“Edit with Emacs” A description for this template.
entry The type, in this case a normal org-mode entry with a headline.
file+datetree The entry will go in a file, specified by the next argument, arranged in a date-tree.
“~/work/org/comments.org” The file into which the entries will be captured.
“* %U\n%i\n\n” The actual template. This one says the entry has a headline (the *), the %U is replaced with a date-time stamp, and %i is replaced with the edited text buffer contents. Each \n is a newline.
:immediate-finish t Don’t clock in a task or prompt me for anything, just capture it and save it.

In the comments.org file, org-capture will automatically create the necessary date-tree outline sections. So if I save a comment today, it will be filed under an outline showing the year, month, and day, with its own date/time-stamp (%U), like this:

* 2015
** 2015-06 June
*** 2015-06-08 Monday
**** [2015-06-08 Mon 21:12]
     A test comment.

Now I just need the code to hook into edit-server and pass the edited buffer contents to this capture template. Here’s the code I added to my .emacs:

(defun ajb/save-edit-server-buffer ()
  "Save the edit-server buffer as an org-mode entry via org-capture
   when it is saved with C-c C-c.
   Should be called from edit-server-done-hook."
  (when (not (string-match "edit-server" (buffer-name)))
    (goto-char (point-min))
    (insert (buffer-name) "\n")
    (mark-whole-buffer)
    (org-capture nil "e")
    (goto-char (point-min))
    (kill-line 1)))

(add-hook 'edit-server-done-hook 'ajb/save-edit-server-buffer)

It’s fairly straightforward, but it took a while to work out the bugs. First, for some reason, the hook gets called twice – once on a buffer named for the web site where the text came from (like “blogspot.com”), then again for a buffer called " edit-server-343434", where the number at the end keeps changing. I’m not sure why it does that, but the (when line handles that by ignoring the buffer with “edit-server” in the name.

The rest is pretty simple. It goes to the beginning of the buffer (goto-char (point-min)), and inserts the name of the buffer and a newline there. I did that so I have the name of the site where the comment came from saved at the beginning of it. It only gives the domain. It might be nice to have the full URL to the page, so I may come back later and see if edit-server has access to that information. But at least this way I know what the comment was about, and I can do a site: search if I need to track down the specific page.

Next it marks the whole buffer for processing by org-capture, which it calls specifying the “e” template that I setup earlier. This is where the actual capture happens, and the buffer is saved in comments.org, in the right spot in the date-tree outline.

Next it goes to the beginning of the buffer again, and removes the first line (kill-line 1), which contains the buffer-name that I wrote there earlier. That way the buffer name is saved to comments.org, but doesn’t show up over in Chromium. In other words, it restores the buffer to the contents it had when the hook was called.

Then the function ends, so edit-server moves on and sends the contents of the buffer back over to Chromium, where I can post it or whatever.

I’ve tested it a bit, and it works nicely. As I mentioned, I’ll probably try to get the full URL instead of just the buffer-name, if possible. Other than that, I can’t think of any improvements right now, but something may come to me as I use it for a while. The main thing I’ll have to get used to is using Edit with Emacs for all my posting, and not typing quick comments in directly if I want to save them.

I’m still hoping to switch to Qupzilla for more of my browsing, and I don’t think there’s an Edit with Emacs extension for it, so as a future project I may need to port that over to it.

Update: I added clocking as a feature here.

Org-Mode: Returning to Previous Task on Clock-out

I’ve been getting into http://orgmode.org|org-mode over the last couple years, using it to organize my work and as many other things as possible. Org-mode is an organizer (and much more) that runs in Emacs, which I use for many other things, so it’s a great fit. I owe a great deal of thanks to <http://doc.norang.ca/org-mode.html|Bernt Hansen, who put his complete org-mode configuration> online with detailed explanation and instructions. He uses it in a very http://gettingthingsdone.com|GTD-style way, which is exactly what I wanted, so I cut-and-pasted much of his base configuration at first, but since then I’ve done some tinkering of my own.

One of the great features of org-mode is clocking. I can clock into any task with C-c C-x C-i (or hitting I on it in the agenda), and then org-mode logs the time I spend on that task until I clock out, either with C-c C-x C-o (or O in the agenda) or by marking the task DONE. Each task has a LOGBOOK which collects all the time spent on it, so I can print invoices, productivity reports, or anything else I want out of that data.

Thanks to some of Bernt’s customizations, once I punch in to start the day, org-mode is always clocking something until I punch out. Anytime I clock-out of one task, it switches to clocking something else, so my time is always accounted for somewhere.

But which task should it switch to? In Bernt’s configuration, when you close one task, it switches to that task’s parent. To explain what a parent task is, here’s a very simplified example org-mode file:

* Garden
** TODO Plant Vegetables
*** NEXT Peas
*** TODO Beans
*** TODO Corn

* Blogging
** TODO Write a post on clocking in org-mode

* TODO Write Novel
** NEXT Write outline
** TODO Write preface
** TODO Write chapter 1
** TODO Write chapter 2

Org-mode is based on an outline type of format. Every line starting with one or more asterisks is a heading, and the number of asterisks determines the level. Here there are three top-level (1 asterisk) headings, and each one has some sub-headings. So “Garden” is the parent of “Plant Vegetables,” which is the parent of “Peas,” “Beans,” and “Corn.” Likewise, “Write Novel” is the parent of “Write outline,” “Write preface,” and so on. (If you’re curious about what TODO and NEXT mean, they both signify tasks that are waiting for me to act on them. The difference is that NEXT tasks are the next one I intend to do for their project, and they shouldn’t have to wait on any other tasks.)

In Bernt’s configuration, when you close one task, the clock switches to that task’s parent. If I’m working on my novel, that might make sense. When I finish the outline, I mark that task done, and the clock switches to its parent, “Write Novel.” I still have some more time to write, so I move down and clock into “Write preface” and continue writing. All the time gets clocked into either one of the sub-tasks or the parent.

However, that doesn’t work so well when I’m not likely to move right from one sub-task to another, which is usually the case for me. For instance, let’s say I’m working on this blog post, so I have that task clocked in. I hear thunder outside, and realize I should run out and plant my peas before the rain starts. I quickly clock into “Peas” (which clocks me out of the blogging task), and run out to plant them. When I come back in and mark “Peas” DONE, I don’t want it to switch to the parent task “Plant Vegetables.” I want it to switch back to the blogging task I was working on before.

I find that that’s usually the way my work-flow goes. I can do that manually with a few keystrokes, but it’d be nicer to have it happen automatically, and not have that few moments clocked into a task I didn’t intend. So I tracked this functionality down to the bh/clock-out-maybe function. Here’s the original:

(defun bh/clock-out-maybe ()
  (when (and bh/keep-clock-running
             (not org-clock-clocking-in)
             (marker-buffer org-clock-default-task)
             (not org-clock-resolving-clocks-due-to-idleness))
    (bh/clock-in-parent-task)))

This checks a few things, and if they all are true, it clocks in the parent task. So I want it to clock in the previously clocked task instead. I changed it to this:

(defun bh/clock-out-maybe ()
  (when (and bh/keep-clock-running
             (not org-clock-clocking-in)
             (marker-buffer org-clock-default-task)
             (not org-clock-resolving-clocks-due-to-idleness))
    (if (marker-buffer org-clock-interrupted-task)
        (org-with-point-at org-clock-interrupted-task
          (org-clock-in))
      (bh/clock-in-parent-task))))

As you can see, it’s not a huge change. When it runs, it first checks to see if there’s a marker for a task that was interrupted by the current task, which org-mode handily keeps track of. If there is, it switches to that mark and clocks in there. If there’s not – if the current task is my first task of the day, for instance – then it uses the previous behavior of switching to the parent. Either way, the clock keeps running.

This works better for me, and I also like the fact that it better matches the way org-mode’s capture mode works, which also switches back to the interrupted task after capturing something. Although it’s a very small change code-wise, it taught me a lot about the internals of org-mode as I tracked this down.