Install gidgetlab and httpx

Install gidgetlab and httpx if you have not already. Using a virtual environment is recommended.

python3 -m pip install gidgetlab[httpx]

Create an issue

We'll use the same example as in the previous post but replace aiohttp with httpx. Copy the following into the file create_issue.py using your favorite editor:

import asyncio
import os
import httpx
import gidgetlab.httpx


async def main():
    async with httpx.AsyncClient() as client:
        gl = gidgetlab.httpx.GitLabAPI(
            client, "gidgetlab", access_token=os.environ.get("GL_ACCESS_TOKEN")
        )
        await gl.post(
            "/projects/beenje%2Fstrange-relationship/issues",
            data={
                "title": "We got a problem",
                "description": "You should use HTTPX!",
            })


asyncio.run(main())

If you check the example with aiohttp from my previous post, you can see it's pretty similar.

$ diff -u aiohttp_create_issue.py create_issue.py
--- aiohttp_create_issue.py 2020-05-31 21:31:52.000000000 +0200
+++ create_issue.py 2020-05-31 21:26:19.000000000 +0200
@@ -1,12 +1,14 @@
 import asyncio
 import os
-import aiohttp
-from gidgetlab.aiohttp import GitLabAPI
+import httpx
+import gidgetlab.httpx


 async def main():
-    async with aiohttp.ClientSession() as session:
-        gl = GitLabAPI(session, "beenje", access_token=os.getenv("GL_ACCESS_TOKEN"))
+    async with httpx.AsyncClient() as client:
+        gl = gidgetlab.httpx.GitLabAPI(
+            client, "gidgetlab", access_token=os.environ.get("GL_ACCESS_TOKEN")
+        )
         await gl.post(
             "/projects/beenje%2Fstrange-relationship/issues",
             data={
@@ -15,5 +17,4 @@
             })


-loop = asyncio.get_event_loop()
-loop.run_until_complete(main()
+asyncio.run(main())

The only real difference is the use of async with httpx.AsyncClient() as client instead of async with aiohttp.ClientSession() as session. asyncio.run() was introduced in Python 3.7 and is the new way to run an async function.

Save the file and run it in the command line after exporting your GitLab access token.

In Unix / Mac OS:

export GL_ACCESS_TOKEN=<your token>

In Windows:

set GL_ACCESS_TOKEN=<your token>

python3 -m create_issue

There should be a new issue created in the strange-relationship project. Check it out: https://gitlab.com/beenje/strange-relationship/issues

Starlette bot

In the same virtual environment as before install Starlette and uvicorn:

python3 -m pip install starlette uvicorn

Save the following in a file named bot.py:

import os
import httpx
import gidgetlab.routing
import gidgetlab.sansio
import gidgetlab.httpx
from starlette.applications import Starlette
from starlette.requests import Request
from starlette.responses import Response
from starlette.routing import Route

router = gidgetlab.routing.Router()


@router.register("Issue Hook", action="open")
async def issue_opened_event(event, gl, *args, **kwargs):
    """Whenever an issue is opened, greet the author and say thanks."""
    url = f"/projects/{event.project_id}/issues/{event.object_attributes['iid']}/notes"
    message = f"Thanks for the report @{event.data['user']['username']}! I will look into it ASAP! (I'm a bot)."
    await gl.post(url, data={"body": message})


async def webhook(request: Request) -> Response:
    """Handler that processes GitLab webhook requests"""
    body = await request.body()
    secret = os.environ.get("GL_SECRET")
    event = gidgetlab.sansio.Event.from_http(request.headers, body, secret=secret)
    async with httpx.AsyncClient() as client:
        gl = gidgetlab.httpx.GitLabAPI(
            client, "gidgetlab", access_token=os.environ.get("GL_ACCESS_TOKEN")
        )
        await router.dispatch(event, gl)
    return Response(status_code=200)


app = Starlette(routes=[Route("/", webhook, methods=["POST"])])

The Issue Hook handler is exactly the same as when using aiohttp. gidgetlab abstracts away the HTTP client used. To implement the bot, the only thing needed is an endpoint to handle webhook POST requests.

Run:

uvicorn --reload bot:app
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [40232] using statreload
INFO:     Started server process [40234]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

That's it. We have an ASGI server running on port 8000 that can receive events from GitLab. We could test it by using ngrok as in the previous post. This time we'll just fake some events using httpie.

Testing with httpie

For testing purpose, let's add a dummy event handler that is easy to trigger.

@router.register("Push Hook")
async def dummy_action_on_push(event, gl, *args, **kwargs):
    print(f"Received {event.event}")
    print("Triggering some action...")
    await gl.sleep(1)
    print("Action done")

In one terminal, run:

uvicorn --reload bot:app

In another one:

http POST 127.0.0.1:8000  "X-Gitlab-Event:Push Hook" Content-Type:application/json

You should see the following output in each respective terminal:

Received Push Hook
Triggering some action...
Action done
INFO:     127.0.0.1:58814 - "POST / HTTP/1.1" 200 OK

HTTP/1.1 200 OK
date: Wed, 27 May 2020 20:39:02 GMT
server: uvicorn
transfer-encoding: chunked

If you want to use a secret you should pass it on both sides:

export GL_SECRET=12345
uvicorn --reload bot:app


http POST 127.0.0.1:8000 x-gitlab-token:12345 "X-Gitlab-Event:Push Hook" Content-Type:application/json

You can see both examples on the following screenshot.

Starlette startup and shutdown events

Starlette can register event handlers to run on startup and shutdown. Instead of creating a new httpx client on every new request, we could re-use the same.

async def create_client() -> None:
    """Startup handler that creates the GitLabAPI instance"""
    client = httpx.AsyncClient()
    app.state.gl = gidgetlab.httpx.GitLabAPI(
        client, "gidgetlab", access_token=os.environ.get("GL_ACCESS_TOKEN")
    )


async def close_client() -> None:
    """Shutdown handler that closes the httpx client"""
    await app.state.gl._client.aclose()


async def webhook(request: Request) -> Response:
    """Handler that processes GitLab webhook requests"""
    body = await request.body()
    secret = os.environ.get("GL_SECRET")
    event = gidgetlab.sansio.Event.from_http(request.headers, body, secret=secret)
    await router.dispatch(event, request.app.state.gl)
    return Response(status_code=200)


app = Starlette(
    routes=[Route("/", webhook, methods=["POST"])],
    on_startup=[create_client],
    on_shutdown=[close_client],
)

In the create_client function, we also store the GitLabAPI instance on the app.state. This allows us to access it using request.app in the request and to close the httpx client on application shutdown.

Background tasks

In the above code, the Response is only sent when all the dispatched event handlers have been executed. Some event handlers might take some time to run if you trigger many actions or you might want to sleep (asyncio.sleep of course not to block the event loop) between different actions. You probably noticed that's actually exactly what I did in my dummy push hook handler.

To illustrate that let's increase the sleep and print the date in our handler:

import datetime


@router.register("Push Hook")
async def dummy_action_on_push(event, gl, *args, **kwargs):
    print(f"Received {event.event}")
    print(f"Triggering some action at {datetime.datetime.utcnow()}...")
    await gl.sleep(5)
    print(f"Action done at {datetime.datetime.utcnow()}")

If we send a Push Hook event, we'll only get a response after 5 seconds. Not great... We can see that the server isn't blocked. We can send several requests and they are all processed in parallel. But the response is only sent after the event handler is done.

Action done is printed before the 200 is sent.

When receiving a webhook, you should send the HTTP response as fast as possible. This is stated in GitLab's documentation: Your endpoint should send its HTTP response as fast as possible. If you wait too long, GitLab may decide the hook failed and retry it.

One way to achieve that would be to use a task queue like Celery or RQ to run the event handlers. I'm actually using RQ in an aiohttp bot I created.

A nice feature of Starlette is that you can attach a background task to a response. We can thus run the dispatch function as a BackgroundTask. This will ensure that the response is sent as soon as the event has been received and parsed:

from starlette.background import BackgroundTask


async def webhook(request: Request) -> Response:
    """Handler that processes GitLab webhook requests"""
    body = await request.body()
    secret = os.environ.get("GL_SECRET")
    event = gidgetlab.sansio.Event.from_http(request.headers, body, secret=secret)
    task = BackgroundTask(router.dispatch, event, request.app.state.gl)
    return Response(status_code=200, background=task)

If we perform the same test as before we see that the event is dispatched only after the response was sent. It doesn't matter how long each handler takes.

Received Push Hook is printed after the 200 is sent.

Of course handlers shouldn't block the event loop! As router.dispatch is an async function, Starlette will just await on it. If an event handler is performing some blocking action, it should be run in a thread or process pool. Otherwise the above code is all that is required.

Better error handling

One thing we didn't pay attention to is error handling. What happens if gidgetlab.sansio.Event.from_http raises an Exception? Starlette will return a 500 (Internal Server Error) HTTP response. That's the proper thing to do. Your endpoint should ALWAYS return a valid HTTP response.

But in the bot logs, we can see that exception. Not very clean.

We should catch those exceptions and handle them properly.

from starlette.responses import Response, PlainTextResponse


async def webhook(request: Request) -> Response:
    """Handler that processes GitLab webhook requests"""
    body = await request.body()
    secret = os.environ.get("GL_SECRET")
    try:
        event = gidgetlab.sansio.Event.from_http(request.headers, body, secret=secret)
    except gidgetlab.HTTPException as e:
        return PlainTextResponse(status_code=e.status_code, content=str(e))
    except gidgetlab.GitLabException as e:
        return PlainTextResponse(status_code=500, content=str(e))
    task = BackgroundTask(router.dispatch, event, request.app.state.gl)
    return Response(status_code=200, background=task)

Much nicer now! Everything is in place for a production ready bot.

Linux

Note that bzip2 is required to run the installation.

curl -LO https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -bp $HOME/miniconda
rm -f Miniconda3-latest-Linux-x86_64.sh
# Let conda update your ~/.bashrc
source $HOME/miniconda/bin/activate
conda init

macOS

curl -LO https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
bash Miniconda3-latest-MacOSX-x86_64.sh -bp $HOME/miniconda
rm -f Miniconda3-latest-MacOSX-x86_64.sh
# Let conda update your ~/.bash_profile
source $HOME/miniconda/bin/activate
conda init

Windows

Download and run the Miniconda3 installer from https://docs.conda.io/en/latest/miniconda.html#windows-installers. To use conda, open the Anaconda Prompt from the start menu.

Configuration

On Linux and macOS, if you don't want conda to activate the base environment by default (and modify your PATH), you should run:

conda config --set auto_activate_base false

This doesn't really apply to Windows as you have to start the Anaconda Prompt to use conda.

To be able to install package from conda-forge, add the conda-forge channel to your configuration. This applies to all platforms.

conda config --add channels conda-forge

Environment creation

Now that we have conda installed and configured, getting epics-base is as easy as running:

conda create -y -n epics epics-base

Note that you don't need any compiler or to install any other packages. The only requirement is conda. As of may 2020, the version installed should be 7.0.3.1.

Environment activation

To start using EPICS, activate the environment:

conda activate epics

You now have access to all the binaries provided by epics-base:

caget -h
pvget -h
softIocPVA
epics> exit

On Windows, there is currently a small issue. If you run softIocPVA -h, you will see that the compiled-in path to softIocPVA.dbd is incorrect:

(epics) C:\Users\IEUser>softIocPVA -h
Usage: softIocPVA [-D softIoc.dbd] [-h] [-S] [-a ascf]
        [-m macro=value,macro2=value2] [-d file.db]
        [-x prefix] [st.cmd]
Compiled-in path to softIocPVA.dbd is:
        D:/bld/epics-base_1588657178544/_h_env/epics/dbd/softIocPVA.dbd

The path is the one that was used when the epics-base conda package was created. Conda usually automatically replaces this $PREFIX variable when creating an environment. It works on Linux and macOS but not on Windows in this case. You have to give the explicit path to the dbd manually. You can use the %EPICS_BASE% environment variable that is automatically set during the activation of the epics environment:

(epics) C:\Users\IEUser>softIocPVA -D %EPICS_BASE%\dbd\softIocPVA.dbd
epics>

Note that if I understand correctly this tech-talk message, next release should use a relative path and remove this issue.

After activation, you can see that several EPICS environment variables have been set. The PATH was also updated. It includes both $CONDA_PREFIX/bin as well as $EPICS_BASE/bin/$EPICS_HOST_ARCH:

(epics) [tux@964ef40cabbb ~]$ env | grep EPICS
EPICS_BASE_HOST_BIN=/home/tux/miniconda/envs/epics/epics/bin/linux-x86_64
EPICS_BASE_VERSION=7.0.3.1
EPICS_BASE=/home/tux/miniconda/envs/epics/epics
EPICS_HOST_ARCH=linux-x86_64
(epics) [tux@964ef40cabbb ~]$ echo $PATH
/home/tux/miniconda/envs/epics/epics/bin/linux-x86_64:/home/tux/miniconda/envs/epics/bin:/home/tux/miniconda/condabin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/tux/.local/bin:/home/tux/bin
(epics) [tux@964ef40cabbb ~]$

Those variables are set by the activation script part of the epics-base package. Running conda deactivate will unset those variables:

(epics) [tux@964ef40cabbb ~]$ conda deactivate
(base) [tux@964ef40cabbb ~]$ env | grep EPICS
(base) [tux@964ef40cabbb ~]$ echo $PATH
/home/tux/miniconda/bin:/home/tux/miniconda/condabin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/tux/.local/bin:/home/tux/bin
(base) [tux@964ef40cabbb ~]$

Simple test

With your editor of choice, create the test.db file that contains:

record(ai, "temperature:water")
{
    field(DESC, "Water temperature in the fish tank")
}

Open a terminal and activate the epics environment.

On Linux and macOS, run:

softIocPVA -d test.db

On Windows, run:

softIocPVA -D %EPICS_BASE%\dbd\softIocPVA.dbd -d test.db

Open another terminal and run:

CI0011906:~ $ conda activate epics
(epics) CI0011906:~ $ caget temperature:water
temperature:water              0
(epics) CI0011906:~ $ caget temperature:water.DESC
temperature:water.DESC         Water temperature in the fish tank
(epics) CI0011906:~ $ caput temperature:water 21
Old : temperature:water              0
New : temperature:water              21
(epics) CI0011906:~ $ caget temperature:water
temperature:water              21
(epics) CI0011906:~ $

The following screenshots show the result on macOS and Windows.

Pre-requisites

The pre-requisites are different depending on the platform.

conda create -y -n epics-dev epics-base make compilers
conda activate epics-dev

xcode-select --install

conda create -y -n epics-dev epics-base make compilers
conda activate epics-dev

(epics-dev) CI0011906:~ $ echo $CONDA_BUILD_SYSROOT
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk

export CONDA_BUILD_SYSROOT=/opt/MacOSX10.9.sdk

export MACOSX_DEPLOYMENT_TARGET=10.9

conda create -n epics-dev epics-base epics-base-static-libs make vs2017_win-64
conda activate epics-dev

IOC creation

Make sure you activated the epics-dev environment you created. Note that we didn't have to specify perl when creating the environment. It's installed with epics-base as run dependency.

On Linux and macOS:

(epics-dev) CI0011906:~ $ mkdir -p $HOME/EPICS/testIoc
(epics-dev) CI0011906:~ $ cd $HOME/EPICS/testIoc
(epics-dev) CI0011906:~/EPICS/testIoc $ makeBaseApp.pl -t example testIoc
(epics-dev) CI0011906:~/EPICS/testIoc $ makeBaseApp.pl -i -t example testIoc
Using target architecture darwin-x86 (only one available)
The following applications are available:
    testIoc
What application should the IOC(s) boot?
The default uses the IOC's name, even if not listed above.
Application name?
(epics-dev) CI0011906:~/EPICS/testIoc $ make
...
(epics-dev) CI0011906:~/EPICS/testIoc $ cd iocBoot/ioctestIoc
(epics-dev) CI0011906:~/EPICS/testIoc/iocBoot/ioctestIoc $ chmod a+x st.cmd
(epics-dev) CI0011906:~/EPICS/testIoc/iocBoot/ioctestIoc $ ./st.cmd
#!../../bin/darwin-x86/testIoc
< envPaths
epicsEnvSet("IOC","ioctestIoc")
epicsEnvSet("TOP","/Users/benjaminbertrand/EPICS/testIoc")
epicsEnvSet("EPICS_BASE","/Users/benjaminbertrand/miniconda3/envs/epics-dev/epics")
cd "/Users/benjaminbertrand/EPICS/testIoc"
## Register all support components
dbLoadDatabase "dbd/testIoc.dbd"
testIoc_registerRecordDeviceDriver pdbbase
## Load record instances
dbLoadTemplate "db/user.substitutions"
dbLoadRecords "db/testIocVersion.db", "user=benjaminbertrand"
dbLoadRecords "db/dbSubExample.db", "user=benjaminbertrand"
#var mySubDebug 1
#traceIocInit
cd "/Users/benjaminbertrand/EPICS/testIoc/iocBoot/ioctestIoc"
iocInit
Starting iocInit
############################################################################
## EPICS R7.0.3.1
## EPICS Base built May  5 2020
############################################################################
iocRun: All initialization complete
## Start any sequence programs
#seq sncExample, "user=benjaminbertrand"
epics> dbl
benjaminbertrand:testIoc:version
benjaminbertrand:xxxExample
benjaminbertrand:circle:step
benjaminbertrand:circle:period
benjaminbertrand:line:b
benjaminbertrand:aiExample
...

On Windows:

(epics-dev) C:\Users\IEUser> mkdir EPICS\testIoc
(epics-dev) C:\Users\IEUser> cd EPICS\testIoc
(epics-dev) C:\Users\IEUser\EPICS\testIoc> perl %EPICS_BASE_HOST_BIN%\makeBaseApp.pl -t example testIoc
(epics-dev) C:\Users\IEUser\EPICS\testIoc> perl %EPICS_BASE_HOST_BIN%\makeBaseApp.pl -i -t example testIoc
Using target architecture windows-x64 (only one available)
The following applications are available:
    testIoc
What application should the IOC(s) boot?
The default uses the IOC's name, even if not listed above.
Application name?
(epics-dev) C:\Users\IEUser\EPICS\testIoc> make
...
(epics-dev) C:\Users\IEUser\EPICS\testIoc> cd iocBoot\ioctestIoc
(epics-dev) C:\Users\IEUser\EPICS\testIoc\iocBoot\ioctestIoc> ..\..\bin\windows-x64\testIoc.exe st.cmd
#!../../bin/windows-x64/testIoc
< envPaths
epicsEnvSet("IOC","ioctestIoc")
epicsEnvSet("TOP","C:/Users/IEUser/EPICS/testIoc")
epicsEnvSet("EPICS_BASE","C:/Users/IEUser/miniconda3/envs/epics-dev/epics")
cd "C:/Users/IEUser/EPICS/testIoc"
## Register all support components
dbLoadDatabase "dbd/testIoc.dbd"
testIoc_registerRecordDeviceDriver pdbbase
## Load record instances
dbLoadTemplate "db/user.substitutions"
dbLoadRecords "db/testIocVersion.db", "user=IEUser"
dbLoadRecords "db/dbSubExample.db", "user=IEUser"
#var mySubDebug 1
#traceIocInit
cd "C:/Users/IEUser/EPICS/testIoc/iocBoot/ioctestIoc"
iocInit
Starting iocInit
############################################################################
## EPICS R7.0.3.1
## EPICS Base built May  5 2020
############################################################################
iocRun: All initialization complete
## Start any sequence programs
#seq sncExample, "user=IEUser"
epics> dbl
IEUser:xxxExample
IEUser:circle:angle
IEUser:line:a
IEUser:circle:x
IEUser:circle:y
IEUser:calcExample
...

We have a running IOC on all 3 platforms!

VScode

We'll first need an editor. I've been a Vim user for many years, but have to say I started to use VScode more lately, with VSCodeVim of course :-). Microsoft is really doing a nice job. There are many great extensions. I can only recommend it.

Download VScode from https://code.visualstudio.com/.

Obviously, an editor is very personal. Pick the one you prefer!

Git

To work with code, Git is essential. Download and install it from https://git-scm.com/downloads.

Microsoft’s Visual C++

To compile native code (C, C++, etc.) on Windows, we need Microsoft’s Visual C++. As explained in this Python wiki, each Python version uses a specific compiler version.

Since CPython 3.5, Visual C++ 14.X is required. This compiler has been part of Visual Studio since Visual Studio 2015.

As of May 2020, the current version of Visual Studio that you can download from https://visualstudio.microsoft.com/downloads/ is Visual Studio 2019, which comes with Visual C++ 14.2.

We could use that version, but conda-forge currently uses Visual Studio 2017. The transition from vs2015 to vs2017 was done in April 2020. Downloading an older release requires a Microsoft account.

Once logged in, go to https://visualstudio.microsoft.com/vs/older-downloads/ and download the Build Tools for Visual Studio 2017. You don't need to download the full Visual Studio edition.

During installation, only select the build tools.

The installation process will take some time. Be patient.

Miniconda3

Now that we have an editor, Git and Windows C++ compilers, the last tool missing is conda. Download and install Miniconda3 from https://docs.conda.io/en/latest/miniconda.html#windows-installers.

To use conda, start the Anaconda Prompt from the Start menu.

Just a few more steps to configure conda.

• Add conda-forge channel:

  conda config --add channels conda-forge

• Install conda-build:

  conda install -y conda-build

• Download the conda_build_config.yaml file from conda-forge-pinning-feedstock under the home directory:

  curl -LO https://raw.githubusercontent.com/conda-forge/conda-forge-pinning-feedstock/master/recipe/conda_build_config.yaml

The conda_build_config.yaml file contains the version of compilers to use as well as the globally pinned packages. Notice that the compiler is set to vs2017 for Windows.

Note that this file contains several versions for Python: 3.6 and 3.7 at the time of writing. This means that when building conda packages with Python, you'll always build 2 packages (except for noarch). You can keep it as is if you want to test every versions. In most cases, testing one version of Python is enough. Especially during development. You can tune that file to your needs. I'll comment out Python 3.6.

python:
#  - 3.6.* *_cpython
  - 3.7.* *_cpython

That's it! We now have all the tools required to build conda packages locally on Windows.

Install gidgetlab and aiohttp

Install gidgetlab and aiohttp if you have not already. Using a virtual environment is recommended.

python3.6 -m pip install gidgetlab[aiohttp]

Create a GitLab Personal Access Token

In order to use GitLab's API, you'll need to create a personal access token that will be used to authenticate yourself to GitLab.

1. Go to https://gitlab.com/profile/personal_access_tokens

   Or, from GitLab, go to your Settings > Access Tokens.

2. Under Name, enter a short description, to identify the purpose of this token. I recommend something like: bot tutorial.

4. Under Scopes, check the api scope.

5. Click Create personal access token. You will see your new personal access token (a 21 characters string). Click on the copy to clipboard icon and and paste it locally in a text file for now. If you have a password manager like 1password, use that.

   This is the only time you'll see this token in GitLab. If you lose it, you'll need to revoke it and create another one.

Store the Personal Access Token as an environment variable

In Unix / Mac OS:

export GL_ACCESS_TOKEN=your token

In Windows:

set GL_ACCESS_TOKEN=your token

Note that these will only set the token for the current process. If you want this value stored permanently, you have to edit the bashrc file.

Create an issue

Open a new file, for example create_issue.py in your favorite editor.

Copy the following into create_issue.py. Instead of "beenje" however, use your own GitLab username:

import asyncio
import os
import aiohttp
from gidgetlab.aiohttp import GitLabAPI

async def main():
    async with aiohttp.ClientSession() as session:
        gl = GitLabAPI(session, "beenje", access_token=os.getenv("GL_ACCESS_TOKEN"))

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

We only instantiate a GitLabAPI class from gidgetlab by passing who we are ("beenje" in this example) and our GitLab personal access token stored in the GL_ACCESS_TOKEN environment variable. Note that to interact with a private GitLab instance, you just have to pass the url to GitLabAPI:

gl = GitLabAPI(session, "beenje", access_token=os.getenv("GL_ACCESS_TOKEN"),
               url="https://mygitlab.example.com")

By default, the url is set to https://gitlab.com.

So let's create an issue in one of my personal repo. Take a look at GitLab's documentation for creating a new issue.

To create an issue, you should make a POST request to the url /projects/:id/issues and supply the parameters title (required) and description. The id can be the project ID or URL-encoded path of the project owned by the authenticated user.

With gidgetlab, this looks like the following:

await gl.post(
    "/projects/beenje%2Fstrange-relationship/issues",
    data={
        "title": "We got a problem",
        "description": "Use more emoji!",
    })

beenje%2Fstrange-relationship is the URL-encoded path of the project. We could have used the id 7898119 instead. The project ID can be found on the project main page.

Add the above code right after you instantiate GitLabAPI. Your file should now look like the following:

import asyncio
import os
import aiohttp
from gidgetlab.aiohttp import GitLabAPI


async def main():
    async with aiohttp.ClientSession() as session:
        gl = GitLabAPI(session, "beenje", access_token=os.getenv("GL_ACCESS_TOKEN"))
        await gl.post(
            "/projects/beenje%2Fstrange-relationship/issues",
            data={
                "title": "We got a problem",
                "description": "Use more emoji!",
            })


loop = asyncio.get_event_loop()
loop.run_until_complete(main())

Feel free to change the title and the body of the message. Save and run it in the command line:

python3.6 -m create_issue

There should be a new issue created in the strange-relationship project. Check it out: https://gitlab.com/beenje/strange-relationship/issues

Comment on issue

Let's try a different exercise, to get ourselves more familiar with GitLab's API. Take a look at GitLab's create a comment documentation: POST /projects/:id/issues/:issue_iid/notes

Leave a comment in the issue you just created:

await gl.post(
    "/projects/beenje%2Fstrange-relationship/issues/1/notes",
    data={"body": "This is a comment"},
)

Replace 1 with the issue number you created.

Close the issue

Let's now close the issue that you've just created.

Take a look at the documentation to edit an issue.

The method for editing an issue is PUT instead of POST, which we've seen in the previous two examples. In addition, to close an issue, you're basically editing an issue, and setting the state_event to close.

Use gidgetlab to close the issue:

await gl.put(
    "/projects/beenje%2Fstrange-relationship/issues/1",
    data={"state_event": "close"},
)

Replace 1 with the issue number you created.

GitLabBot

gidgetlab actually provides a GitLabBot class to easily create an aiohttp web server that reponds to GitLab webhooks.

Save the following in a file named bot.py:

from gidgetlab.aiohttp import GitLabBot

bot = GitLabBot("beenje")


if __name__ == "__main__":
    bot.run()

And run:

python3 bot.py
======== Running on http://0.0.0.0:8080 ========
(Press CTRL+C to quit)

That's it. You have an aiohttp web server running on port 8080. Of course, it won't do that much. You'll have to register some events if you want the bot to perform some actions. We'll see that later.

Webhook events

When an event is triggered in GitLab, GitLab can notify you about the event by sending a POST request along with the payload.

Some example events are:

• Issues events: any time an issue is created or an existing issue was updated/closed/reopened

• Push events: when you push to the repository except when pushing tags

• Tag events: when you create (or delete) tags to the repository

• Build events: triggered on status change of a Build

The complete list of events is listed here.

Since GitLab needs to send you POST requests for the webhook, you should have a service running somewhere that GitLab can reach. That's usually not on your laptop.

GitHub bot tutorial describes how to deploy your webservice to Heroku. Heroku is a platform as a service and makes it easy to deploy and run your app in the cloud. There are alternatives and you can of course use on your own servers if you want.

For testing purpose, you can actually use your own laptop thanks to ngrok.

Ngrok

Ngrok exposes local servers behind NATs and firewalls to the public internet over secure tunnels. It's an easy way to test locally a webservice.

Check the installation instructions from the website. Note that for simple tests, you don't have to register an account.

If you have a webserver running locally on port 8080, you can expose it by running:

ngrok http 8080

Something similar will appear:

ngrok by @inconshreveable                                       (Ctrl+C to quit)

Session Status                online
Session Expires               7 hours, 59 minutes
Version                       2.2.8
Region                        United States (us)
Web Interface                 http://127.0.0.1:4040
Forwarding                    http://fb7fec7c.ngrok.io -> localhost:8080
Forwarding                    https://fb7fec7c.ngrok.io -> localhost:8080

You can access your local webservice using HTTP and even HTTPS!

curl -X GET https://fb7fec7c.ngrok.io

This address can be accessed from anywhere!. You could give it to a friend or use it as a GitLab webhook.

Ngrok even gives you a web interface on the port 4040 that allows you to inspect all the requests made to the service. Just open http://127.0.0.1:4040 in your browser.

If your bot is still running and you tried to send a GET, you should get a 405 as reply. Only POST methods are handled by the bot.

If you don't have any service listening on port 8080 and try to access the URL given by ngrok, you'll get a 502.

Add the GitLab Webhook

Now that we have a local webservice that can receive requests thanks to ngrok, let's create a webhook on GitLab. If you haven't done so yet, create your own project on GitLab.

Go to your project settings and select Integrations to create a webhook:

• In the URL field, enter the ngrok URL you got earlier.

• For security reasons, type in some random characters under Secret Token (you can use Python secrets.token_hex(16) function)

• Under Trigger, select Issues events, Comments and Merge request events

• Leave Enable SSL verification enabled

• Click Add webhook

Update the Config Variables in your environment

First, export the secret webhook token you just created:

export GL_SECRET=<secret token>

Then, if not already done, export your GitLab personal access token:

export GL_ACCESS_TOKEN=<acess token>

Your first GitLab bot!

Let's start with a bot that responds to every newly created issue in your project. For example, whenever someone creates an issue, the bot will automatically say something like: "Thanks for the report, @user. I will look into this ASAP!"

To respond to webhooks events, we have to register a coroutine using the @bot.router.register decorator:

@bot.router.register("Issue Hook", action="open")
async def issue_opened_event(event, gl, *args, **kwargs):
    pass

In this example we subscribe to the GitLab Issue Hook events, and more specifically to the "open" issues event.

The two important parameters here are: event and gl.

• event here is the representation of GitLab's webhook event. We can access the event payload by doing event.data.

• gl is the gidgetlab GitLabAPI instance, which we can use to make API calls to GitLab, as in the first section.

We already saw that to create a comment on an issue, we need to send: POST /projects/:id/issues/:issue_iid/notes.

Let's look at the Issues events payload to see how we can retrieve the required information:

{
  "object_kind": "issue",
  "user": {
    "name": "Administrator",
    "username": "root",
    "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon"
  },
  "project": {
    "id": 1,
    "name":"Gitlab Test",
    "description":"Aut reprehenderit ut est.",
    "web_url":"http://example.com/gitlabhq/gitlab-test",
    "avatar_url":null,
    "git_ssh_url":"git@example.com:gitlabhq/gitlab-test.git",
    "git_http_url":"http://example.com/gitlabhq/gitlab-test.git",
    "namespace":"GitlabHQ",
    ...
  },
  "repository": {
    "name": "Gitlab Test",
    "url": "http://example.com/gitlabhq/gitlab-test.git",
    "description": "Aut reprehenderit ut est.",
    "homepage": "http://example.com/gitlabhq/gitlab-test"
  },
  "object_attributes": {
    "id": 301,
    "title": "New API: create/update/delete file",
    ...
    "state": "opened",
    "iid": 23,
    "url": "http://example.com/diaspora/issues/23",
    "action": "open"
  },
  ...
}

The project id can be retrieved as event.data["project"]["id"]. As this is quite common, gidgetlab procures a project_id property to access it directly: event.project_id.

To get the issue id, we can use event.data["object_attributes"]["iid"]. Again as accessing event.data["object_attributes"] is quite common, we can use the object_attributes property: event.object_attributes["iid"].

The url to use is thus:

url = f"/projects/{event.project_id}/issues/{event.object_attributes['iid']}/notes"

To greet the author, we have to retrieve the username from the event: event.data["user"]["username"]

Open your bot.py file and add the following coroutine to be called when a new issue is opened:

@bot.router.register("Issue Hook", action="open")
async def issue_opened_event(event, gl, *args, **kwargs):
    """Whenever an issue is opened, greet the author and say thanks."""
    url = f"/projects/{event.project_id}/issues/{event.object_attributes['iid']}/notes"
    message = f"Thanks for the report @{event.data['user']['username']}! I will look into it ASAP! (I'm a bot)."
    await gl.post(url, data={"body": message})

The full file should look like:

from gidgetlab.aiohttp import GitLabBot

bot = GitLabBot("beenje")


@bot.router.register("Issue Hook", action="open")
async def issue_opened_event(event, gl, *args, **kwargs):
    """Whenever an issue is opened, greet the author and say thanks."""
    url = f"/projects/{event.project_id}/issues/{event.object_attributes['iid']}/notes"
    message = f"Thanks for the report @{event.data['user']['username']}! I will look into it ASAP! (I'm a bot)."
    await gl.post(url, data={"body": message})


if __name__ == "__main__":
    bot.run()

Run:

python3 bot.py

Go to your project and open an issue. Wait a few seconds and refresh the page. You should see a new comment added to the issue!

Congrats! You wrote your first GitLab bot!

Of course, using ngrok on your laptop was for testing only. To use it in production, you should deploy it to a server or the cloud. You can check the GitHub bot tutorial to see how to deploy your webservice to Heroku.

REST API using aiohttp

There are already several asyncio web frameworks to build an HTTP server. I decided to go with aiohttp which is kind of the default one.

Using this tutorial I wrote a simple REST API using aiohttp. It uses JSON Web Tokens which is something else I have been wanted to try.

The API has only 3 endpoints:

def setup_routes(app):
    app.router.add_get('/', index)
    app.router.add_post('/login', login)
    app.router.add_post('/messages', post_message)

• / to check that our token is valid

• /login to login

• /messages to post messages

async def login(request):
    config = request.app['config']
    data = await request.json()
    try:
        user = data['username']
        passwd = data['password']
    except KeyError:
        return web.HTTPBadRequest(reason='Invalid arguments')
    # We have only one user hard-coded in the config file...
    if user != config['username'] or passwd != config['password']:
        return web.HTTPBadRequest(reason='Invalid credentials')
    payload = {
        'user_id': 1,
        'exp': datetime.datetime.utcnow() + datetime.timedelta(seconds=config['jwt_exp_delta_seconds'])
    }
    jwt_token = jwt.encode(payload, config['jwt_secret'], config['jwt_algorithm'])
    logger.debug(f'JWT token created for {user}')
    return web.json_response({'token': jwt_token.decode('utf-8')})


@login_required
async def post_message(request):
    if request.content_type != 'application/json':
        return web.HTTPBadRequest()
    data = await request.json()
    try:
        message = data['message']
    except KeyError:
        return web.HTTPBadRequest()
    logger.debug(f'Message received from {request.user}: {message}')
    return web.json_response({'message': message}, status=201)


@login_required
async def index(request):
    return web.json_response({'message': 'Welcome to LegoMac {}!'.format(request.user)})

Raspberry Pi GPIO and asyncio

The default Python package to control the Raspberry Pi GPIO seems to be RPi.GPIO. That's at least what is used in the ImageDemoButton.py from Embedded Artists.

An alternative is the pigpio library which provides a daemon to access the Raspberry Pi GPIO via a pipe or socket interface. And someone (Pierre Rust) already created an aysncio based Python client for the pigpio daemon: apigpio.

Exactly what I needed! It's basically a (incomplete) port of the original Python client provided with pigpio, but far sufficient for my need. I just want to get a notification when pressing the button on top of the screen.

There is an example how to achieve that: gpio_notification.py.

E-paper display and asyncio

The last remaining piece is to make the e-paper display play nicely with asyncio.

The EPD driver uses the fuse library. It allows the display to be represented as a virtual directory of files. So sending a command consists of writing to a file.

There is a library to add file support to asyncio: aiofiles. The only thing I had to do was basically to wrap the file IO in EPD.py with aiofiles:

async def _command(self, c):
    async with aiofiles.open(os.path.join(self._epd_path, 'command'), 'wb') as f:
        await f.write(c)

You can't use await in a class __init__ method. So following some recommendations from stackoverflow, I used the factory pattern and moved the actions requiring some IO to a classmethod:

@classmethod
async def create(cls, *args, **kwargs):
    self = EPD(*args, **kwargs)
    async with aiofiles.open(os.path.join(self._epd_path, 'version')) as f:
        version = await f.readline()
        self._version = version.rstrip('\n')
    async with aiofiles.open(os.path.join(self._epd_path, 'panel')) as f:
        line = await f.readline()
        m = self.PANEL_RE.match(line.rstrip('\n'))
        if m is None:
            raise EPDError('invalid panel string')
        ...

To create an instance of the EPD class, use:

epd = await EPD.create([path='/path/to/epd'], [auto=boolean])

Running the clock as a background task

For the clock, I adapted the clock demo from Embedded Artists repository.

As described in aiohttp documentation I created a background task to display the clock every minute:

async def display_clock(app):
    """Background task to display clock every minute"""
    clock = Clock(app['epd'])
    first_start = True
    try:
        while True:
            while True:
                now = datetime.datetime.today()
                if now.second == 0 or first_start:
                    first_start = False
                    break
                await asyncio.sleep(0.5)
            logger.debug('display clock')
            await clock.display(now)
    except asyncio.CancelledError:
        logger.debug('display clock cancel')


async def start_background_tasks(app):
     app['epd'] = await EPD.create(auto=True)
     app['clock'] = app.loop.create_task(display_clock(app))


async def cleanup_background_tasks(app):
    app['clock'].cancel()
    await app['clock']


def init_app():
    """Create and return the aiohttp Application object"""
    app = web.Application()
    app.on_startup.append(start_background_tasks)
    app.on_cleanup.append(cleanup_background_tasks)
    ...

Stop the clock and display a message

When receiving a message, I first cancel the clock background task and send the messages to the e-paper display using ensure_future so that I can return a json response without having to wait for the message to be displayed as it takes about 5 seconds:

@login_required
async def post_message(request):
    if request.content_type != 'application/json':
        return web.HTTPBadRequest()
    data = await request.json()
    try:
        message = data['message']
    except KeyError:
        return web.HTTPBadRequest()
    # cancel the display clock
    request.app['clock'].cancel()
    logger.debug(f'Message received from {request.user}: {message}')
    now = datetime.datetime.now(request.app['timezone'])
    helpers.ensure_future(request.app['epd'].display_message(message, request.user, now))
    return web.json_response({'message': message}, status=201)

Start the clock when pressing the button

To be able to restart the clock when pressing the button, I connect to the pigpiod when starting the app (in start_background_tasks) and register the on_input callback:

async def start_background_tasks(app):
    app['pi'] = apigpio.Pi(app.loop)
    address = (app['config']['pigpiod_host'], app['config']['pigpiod_port'])
    await app['pi'].connect(address)
    await app['pi'].set_mode(BUTTON_GPIO, apigpio.INPUT)
    app['cb'] = await app['pi'].add_callback(
            BUTTON_GPIO,
            edge=apigpio.RISING_EDGE,
            func=functools.partial(on_input, app))
    ...

In the on_input callback, I re-create the clock background task but only if the previous task is done:

def on_input(app, gpio, level, tick):
    """Callback called when pressing the button on the e-paper display"""
    logger.info('on_input {} {} {}'.format(gpio, level, tick))
    if app['clock'].done():
        logger.info('restart clock')
        app['clock'] = app.loop.create_task(display_clock(app))

traefik

There are no official Traefik docker images for arm yet, but an issue is currently opened. So it should arrive soon!

In the meantime I created my own:

FROM arm32v6/alpine:3.6

RUN apk --update upgrade \
  && apk --no-cache --no-progress add ca-certificates \
  && apk add openssl \
  && rm -rf /var/cache/apk/*

RUN wget -O /usr/local/bin/traefik https://github.com/containous/traefik/releases/download/v1.3.3/traefik_linux-arm \
  && chmod a+x /usr/local/bin/traefik

ENTRYPOINT ["/usr/local/bin/traefik"]

pigpiod

For pigpiod, I first created an image based on arm32v6/alpine but I noticed I couldn't send a SIGTERM to the daemon to stop it properly... I'm not sure why. Alpine being based on musl instead of glibc might be the problem. Here is the Dockerfile I tried:

FROM arm32v6/alpine:3.6

RUN apk add --no-cache --virtual .build-deps \
  gcc \
  make \
  musl-dev \
  tar \
  && wget -O /tmp/pigpio.tar abyz.co.uk/rpi/pigpio/pigpio.tar \
  && tar -xf /tmp/pigpio.tar -C /tmp \
  && sed -i "/ldconfig/d" /tmp/PIGPIO/Makefile \
  && make -C /tmp/PIGPIO \
  && make -C /tmp/PIGPIO install \
  && rm -rf /tmp/PIGPIO /tmp/pigpio.tar \
  && apk del .build-deps

EXPOSE 8888

ENTRYPOINT ["/usr/local/bin/pigpiod", "-g"]

I even tried using tini as entrypoint without luck. So if someone as the explanation, please share it in the comments.

I tried with resin/rpi-raspbian image and I got it working properly right away:

FROM resin/rpi-raspbian:jessie

RUN apt-get update \
  && apt-get install -y \
     make \
     gcc \
     libc6-dev \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

RUN curl -o /tmp/pigpio.tar abyz.co.uk/rpi/pigpio/pigpio.tar \
  && tar -xf /tmp/pigpio.tar -C /tmp \
  && make -C /tmp/PIGPIO \
  && make -C /tmp/PIGPIO install \
  && rm -rf /tmp/pigpio.tar /tmp/PIGPIO

EXPOSE 8888

ENTRYPOINT ["/usr/local/bin/pigpiod", "-g"]

Note that the container has to run in privileged mode to access the GPIO.

aiolegomac

For the main application, the Dockerfile is quite standard for a Python application:

FROM resin/raspberry-pi-python:3.6

RUN apt-get update \
  && apt-get install -y \
     fonts-liberation \
     fonts-dejavu  \
     libjpeg-dev \
     libfreetype6-dev \
     libtiff5-dev \
     liblcms2-dev \
     libwebp-dev \
     zlib1g-dev \
     libyaml-0-2 \
  && apt-get autoremove \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY requirements.txt /app/requirements.txt
RUN python -m venv /opt/legomac \
  && /opt/legomac/bin/pip install -r requirements.txt

COPY . /app

ENTRYPOINT ["/opt/legomac/bin/python"]
CMD ["run.py"]

What about the EPD driver? As it uses libfuse to represent the e-paper display as a virtual directory of files, the easiest was to install it on the host and to mount it as a volume inside the docker container.

HTTPie

To test the server you can of course use curl but I really like HTTPie. It's much more user friendly.

Let's try to access our new server:

$ http GET https://myhost.example.com
HTTP/1.1 401 Unauthorized
Content-Length: 25
Content-Type: application/json; charset=utf-8
Date: Sun, 16 Jul 2017 06:22:42 GMT
Server: Python/3.6 aiohttp/2.2.3

{
    "error": "Unauthorized"
}

Good, we need to login:

$ http POST https://myhost.example.com/login username=john password=foo
HTTP/1.1 400 Bad Request
Content-Length: 32
Content-Type: application/json; charset=utf-8
Date: Sun, 16 Jul 2017 06:18:39 GMT
Server: Python/3.6 aiohttp/2.2.3

{
    "error": "Invalid credentials"
}

Oops, wrong password:

$ http POST https://myhost.example.com/login username=john password='mypassword'
HTTP/1.1 200 OK
Content-Length: 134
Content-Type: application/json; charset=utf-8
Date: Sun, 16 Jul 2017 06:21:14 GMT
Server: Python/3.6 aiohttp/2.2.3

{
    "token": "eyK0eXAiOiJRV5QiLCJhbGciOiJIUzI1NiJ9.eyJ1c3VyX2lkIjoxLCJleHAiOjE1MDB5MTIwOTh9.hECnj4u2mxvZ2r8IEC-db1T-eKTplM4kWJKZoHhtLxQ"
}

We got a token that we can use:

$ http GET https://myhost.example.com 'Authorization: eyK0eXAiOiJRV5QiLCJhbGciOiJIUzI1NiJ9.eyJ1c3VyX2lkIjoxLCJleHAiOjE1MDB5MTIwOTh9.hECnj4u2mxvZ2r8IEC-db1T-eKTplM4kWJKZoHhtLxQ'
HTTP/1.1 200 OK
Content-Length: 43
Content-Type: application/json; charset=utf-8
Date: Sun, 16 Jul 2017 06:22:25 GMT
Server: Python/3.6 aiohttp/2.2.3

{
    "message": "Welcome to LegoMac john!"
}

Authentication is working, so we can send a message:

$ http POST https://myhost.example.com/messages message='Hello World!' 'Authorization: eyK0eXAiOiJRV5QiLCJhbGciOiJIUzI1NiJ9.eyJ1c3VyX2lkIjoxLCJleHAiOjE1MDB5MTIwOTh9.hECnj4u2mxvZ2r8IEC-db1T-eKTplM4kWJKZoHhtLxQ'
HTTP/1.1 201 Created
Content-Length: 27
Content-Type: application/json; charset=utf-8
Date: Sun, 16 Jul 2017 06:23:46 GMT
Server: Python/3.6 aiohttp/2.2.3

{
    "message": "Hello World!"
}

Message sent! HTTPie is nice for testing, but we can make a small script to easily send messages from the command line.

requests

requests is of course the HTTP library to use in Python.

So let's write a small script to send messages to our server. We'll store the server url and username to use in a small yaml configuration file. If we don't have a token yet or if the saved one is no longer valid, the script will retrieve one after prompting us for a password. The token is saved in the configuration file for later use.

The following script could be improved with some nicer error messages by catching exceptions. But it does the job:

import os
import click
import requests
import yaml


def get_config(filename):
    with open(filename) as f:
        config = yaml.load(f)
    return config


def save_config(filename, config):
    with open(filename, 'w') as f:
        yaml.dump(config, f, default_flow_style=False)


def get_token(url, username):
    password = click.prompt('Password', hide_input=True)
    payload = {'username': username, 'password': password}
    r = requests.post(url + '/login', json=payload)
    r.raise_for_status()
    return r.json()['token']


def send_message(url, token, message):
    payload = {'message': message}
    headers = {'Authorization': token}
    r = requests.post(url + '/messages', json=payload, headers=headers)
    r.raise_for_status()


@click.command()
@click.option('--conf', '-c', default='~/.pylegomac.yml',
              help='Configuration file [default: "~/.pylegomac.yml"]')
@click.argument('message')
@click.version_option()
def pylegomac(message, conf):
    """Send message to aiolegomac server"""
    filename = os.path.expanduser(conf)
    config = get_config(filename)
    url = config['url']
    username = config['username']
    if 'token' in config:
        try:
            send_message(url, config['token'], message)
        except requests.exceptions.HTTPError as err:
            # Token no more valid
            pass
        else:
            click.echo('Message sent')
            return
    token = get_token(url, username)
    send_message(url, token, message)
    config['token'] = token
    save_config(filename, config)


if __name__ == '__main__':
    pylegomac()

Let's first create a configuration file:

$ cat ~/.pylegomac.yml
url: https://myhost.example.com
username: john

Send a message:

$ python pylegomac.py 'Beautiful is better than ugly. Explicit is better than implicit. Simple is better than complex. Complex is better than complicated.'
Password:
Message sent

Sending a new message won't request the password as the token was saved in the config file.