your daily cup of tea™

powered by

This month in Citybikes #202510

Leave a comment

It’s been a slow month, but here’s another take at the monthly changelog!

Home Assistant and Citybikes

Ever since HA 0.49 (that was long ago, 2017!), home assistant ships with a Citybikes integration. Over time the project has evolved in architecture and, over time, the integration has become a legacy piece of code. Legacy, in this case, means it cannot be updated with new features until it conforms to a minimum set of requirements. There’s been multiple attempts to add information about electric bikes on the integration over time (#82945 #85279 #96764) but the result was always the same. Any integration must use a third party library to access a remote API.

I think Home Assistant in Citybikes is a very cool thing to have, so it was time to at least do the first steps into bringing out the integration out of legacy.

Back in 2016 I wrote a minimal python client, python-citybikes for the citybikes api to use in cmdbikes. It’s not a particularly well done library, and I believe some parts leave a bit to be desired, but overall it works ok. To use it in HA, I still needed to bring the library back to the future of 2025: asyncio support and being at it, a workflow to publish it to pypi.

Being a bit lazy with re-writing the library, I went with adding a separate asyncio module that maintains the same interface. I have seen other projects take this path and I think it’s the most straightforward way of doing it without worrying about the colour of your functions. I might go back at it to make the library try to be less smart, maybe in 10 years :)

With this, I could work a minimal pull request to Home Assistant: https://github.com/home-assistant/core/pull/151009

Removed this blocker, I hope the Citybikes integration in HA will keep evolving, and as far as I know, that’s already in the works. So, cheers to that!

Quick way to scaffold a new pybikes integration

While working on HA, I noticed their scaffolding script for new integrations and I kind of liked it as a way to quick start things. On pybikes, the approach is to copy paste a system implementation that looks sane and go with that. A problem with this is that it replicates patterns that might not be the best way of doing things (which at the same time means reviews get extra scrutiny).

Inspired by the success of HA on getting integrations I added a scaffolding script to pybikes, which is now the recommended way of starting things. On being run, it generates a valid implementation of a made up system with a static data feed.

$ python -m utils.scaffold example

================================================
Here is your 'Example' implementation

System: pybikes/example.py
Data: pybikes/data/example.json

Run tests by:
$ pytest -k Example

Visualize result:
$ make map! T_FLAGS+='-k Example'

Happy hacking :)
================================================

Being this no replacement for proper documentation, I still believe in code that documents itself, not by docstrings, but by being simple and clear. Of course, that might only work for the simple cases, but at least that’s my spirit.

Open Transport Community Conference 2025

I spent some days in Vienna to attend the first edition of the Open Transport Community Conference, the (un)conference about Free Software and Open Data for mobility and public transport.

We had a session about integrating GBFS information from different sources and looking at open source projects that provide aggregated feeds, being Citybikes one of them. In the session I got to meet other feed aggregators like https://derp.si/ providing feeds from Slovenia, and https://mobidata-bw.de/ from the Baden-Württemberg state and MobilityData. The discussion was also joined by MOTIS and Transitous, a FOSS routing engine and service. MOTIS supports GBFS, and Transitous is interested in integrating more bike share information feeds on their multi modal routing. We are going to work towards Transitous integrating feeds from our GBFS endpoint for the cases where there’s no official GBFS feed available. Transitous might end up being the de-facto routing service for most FOSS transport apps, so it would be amazing for Citybikes to be part of that.

I integrated the feeds from https://gbfs.derp.si/providers in pybikes (#859), meaning we now have pretty good coverage in Slovenia!

Other than that I was very amused to find that Wurstelprater, the amusement park in Prater, has the doors open at night. The first day I arrived late to Vienna, so on my way to find a Würstelstand before going to bed I found myself inside the park, surrounded by shut down attractions, lights, and creepy clown statues. Sneaking in an amusement park was on my todo-bucket list of things I wanted to do, and it turned out there was no sneaking in needed at all!

Duckdb WASM + time-series bike share data demo

During the conference I wanted to show around the historical bike share data Citybikes publishes as parquet files, so I worked a bit more on a demo I have been writing that uses DuckDB-wasm.

The big brain idea here is to build a dashboard that runs entirely on the client using only static assets. Duckdb runs on the client using WebAssembly and can run queries against the parquet files that citybikes hosts.

Check out the demo if you like (the code is hell, first load might take a while to download the duckdb wasm payload. If it fails to load, try again on an incognito). The demo can load any (small) parquet file from Citybikes as a query parameter:

My plan is to integrate these visualizations directly into https://data.citybik.es so parquet files can be explored before downloading. It still has some gotchas though, like the first load and properly catching the wasm payload or a limit on allocated memory, meaning it does not work for any parquet file that allocates more than 1 Gb.

This is all in a preliminary state, but I am really excited that such technology is available. None of this runs with any server process munching data, this is huge!

This month in Citybikes #202509

Leave a comment

Years ago I used to post on twitter every new thing going on in Citybikes. But twitter is a wasteland. Mastodon is cool, but social networks nowadays are meant to be consumed on the instant, so whatever gets posted there matters for a week and then gets buried in the noise.

I usually write long form posts on this blog and whenever I wanted to write something about Citybikes I got a writer’s block thinking on the sheer size of what I wanted to write and how far away in time I need to go to recap. It’s been 15 years of running this project. For example, I would love to write a post about the historical bike share datasets we are now publishing as parquet files, or the technical details behind the new progressive web app.

For a change, let’s try something simple: a monthly changelog. I might still write a post about something in particular from time to time if I feel like it. But the idea is to keep these on point and straightforward so I can come back to these later and reflect. Not that it really matters on today’s internet, these are more for me than for you.

Without further ado

Increasing pybikes health

All the action this month has gone working on pybikes. The general plan was to bring the health of the python library up. Health is the ratio between working systems and failing systems.

Ideally it will be at 100% health, but anything around 90% is acceptable. We can’t control bike share systems shutting down or changing their implementation. But having systems that are outdated does not help anybody. For example, right now it is at 95%, with 32 of the 798 supported systems failing. These reports are generated when tests run.

pybikes health report

Automated syncs with nextbike

Nextbike is a german provider of bike share systems, with operations in many countries. We have had support for nextbike since 2014, making it one of the oldest implementations surviving the passage of time in pybikes.

Being a large provider, nextbike creates and shuts down many systems. Sometimes these are systems that are in a testing phase with a city council, and they are scrapped off after a year. I used to get very anxious about pull requests adding nextbike systems to pybikes. First, because nobody is really using this data for anything useful. Second, because I know they are going to become a legacy that I will need to remove at some point. And finally, because these are usually large and difficult to review. I like to think of the networks we support at Citybikes as a curated list of interesting systems. Going for adding more systems just for the sake of having higher numbers does not make much sense to me. But after time I have come to realize this is just my perspective from the other side as a maintainer. For the rest of the people, the more networks supported, the better.

Nextbike publishes a feed with all their data: http://maps.nextbike.net/maps/nextbike-live.json, and this feed can be used to add instances to pybikes. Pybikes expects instances to live in a json file, so we can’t do any server side processing of this file to generate the instances on runtime. The contract with pybikes is that it is a library that works locally with instances declared on the meta json files, curated by the community. If you want to change the name of a bike share system, you just edit one of these meta files and submit a pull request. text + git is our interface and this is how I like it.

How can we keep this thing both maintainable and healthy?

It’s relatively easy to generate the instances based on the nextbike feed and I have had different forms of such a script for years. The problem is when you want to go for correctness and for such case, there’s no perfect script that will cover all the corner cases. This time I have sorted it out by writing a script that gets you half the way there. And, surprisingly, it’s more than enough. Most of the work needed was on changing the feed to conform to certain conditions, so the script can work with it.

Every morning a workflow runs the script and creates a pull request with the changes (if any). I can go then to that PR and review it or make changes if necessary. No more broken nextbike systems!

My main takeaway from this is to not always try to aim to full correctness. If you told me years ago this could be automated, I would have called you a fool. And today, I would call you a fool if you told me I need to write these by hand. I might still need to review these by hand, edit their name or the name of their city, but it’s significantly less work to maintain now.

Github actions hate

I entered a rabbit hole while writing the github workflow though. I am using https://github.com/peter-evans/create-pull-request action to generate the pull request which really is the most straightforward way of doing it (if you want to use github actions, that is). By default, it uses GITHUB_TOKEN and when using it, it does not trigger additional workflows on the pull request. Taken from the horses mouth:

When you use the repository’s GITHUB_TOKEN to perform tasks, events triggered by the GITHUB_TOKEN, with the exception of workflow_dispatch and repository_dispatch, will not create a new workflow run. This prevents you from accidentally creating recursive workflow runs. For example, if a workflow run pushes code using the repository’s GITHUB_TOKEN, a new workflow will not run even when the repository contains a workflow configured to run when push events occur. For more information, see Use GITHUB_TOKEN for authentication in workflows.

The alternative is to create a Github app to generate these tokens, or create a personal access token with fine grained permissions and impersonate the bot. At that point my head was already hurting, so I went for a PAT with fine grained permissions for now, which I guess is still a huge security liability. This workflow only runs on master, so assumedly I do not need to think on the risks of this running on forks and only members with repo access could leak it. I guess, tokens be tokens, and for anything serious I should be running this on my servers instead. But my general impression with github actions always ends up the same. Things are easy until they are not.

Russian systems

None of the systems we support in Russia worked, so I removed them 8d5a9b and 81cda9. For a while, the problem was the servers were rejecting non-russian ips (this could be easily solved by using a proxy in Russia, but I really do not want to pay for exit nodes). This time, they changed implementation and while looking into it I was surprised they use supabase as the backend. Speaking of tokens, I really hope this one on their app has been deactivated and it’s using some kind of auth after signing in or something else, because if not, woops.

Bringing back support for systems in Finland

There’s open data feeds available from https://digitransit.fi/. They were one of the first to create an open data portal for transport feeds, using OpenTripPlanner (OTP). Back in April they deprecated the feeds using OTP v1. This meant updating the otp parser to use the graphql feeds from their OTP v2 instance.

We only supported Helsinki, but being that their feed covers more cities we added them too. It includes feeds from freebike, donkeyrepublic and smoove. Of interest, some of the systems added use dockless bikes, so this was a good opportunity to test our own implementation of free floating bikes on pybikes. These are the full changes https://github.com/eskerda/pybikes/pull/848

Sometimes adding feeds from open data portals instead of the implementers directly feels weird. Both freebike and donkeyrepublic offer their feeds as GBFS, so we could go for that instead. What’s actually happening is that we are taking a feed that has been converted between different formats and specifications (implementor (GBFS) -> open data portal (otp) -> citybikes). Given that we also offer a gbfs endpoint now we have gone full circle! But when available, it’s always a good idea to take feeds from open data portals, since at least we know its license.

The cities we are covering are: Helsinki, Lappeenranta, Kotka, Kouvola, Turku, Kuopio, Lahti and Tampere.

In case you are curious what a test report ends up looking like, here’s a sample for the digitransit run.

Closing remarks

So I think this was all the major changes this month. Keeping health up on pybikes is a long-running never ending task, but this year we have managed to bring it up from 80% (156 failing systems) to 95% (32 failing systems, which is a cute number). We have removed lots of systems and added some, and hopefully we pass the 800 mark before the end of the year. Huge thanks to the many contributors that made this possible, and without doubt to NLnet for their ongoing support!

This post has turned out rather long, so I hope I can keep it up next month :)

Measure and display FPS on MapLibre GL JS

Leave a comment

If you are working on maps, sooner or later you will want to debug if anything in your code is affecting the performance of the rendering. It’s as simple as monkey patching Map._render and using mrdoob‘s ubiquitous stats.js monitor.

First install the module

$ npm i stats.js

Then add this snippet after initializing the map

import Stats from 'stats.js'
[...]
const map = new maplibregl.Map(...)

const stats = new Stats()
const orig = map._render
map._render = (...args) => {
  stats.begin()
  const res = orig.apply(map, args)
  stats.end()

  return res
}

document.body.appendChild(stats.dom)
[...]

Everyone: please stop using “our most” to announce your products.

1 Comment

I do not know who started this trend (possibly apple but don’t quote me on that), or why it makes me so angry — I think it is because starting phrases with “our” puts in a familiar tone that makes you unconsciously engage, despite not coming from a mom & pop soup shop. Then comes the superlative: most + over-the-top adjective.

Essentially, this is the pattern:

<product name>: Our most <adjective> <thing> [yet|ever].

I am going to keep updating this post with a screenshot every time I see one in the wild.

our our our our our our our our our our our our our our our our our our

An Architecture of One’s Own

Leave a comment

One day I attended a talk about multi-agent systems communicating through ZMQ. That same day I decided I would use it to build the next version of the systems running Citybikes. I felt it was an elegant solution to a problem I had when trying to add bike-share historics. Using agents I could plug in and out of the project without affecting the overall stability. The API would be one agent, historics would be another agent, an old version of the API would be another agent, and so on.

I dreamt of bringing up an agent that would connect to a publisher spitting out json through a socket and graphing bike share availability in real time, for no practical reason whatsoever other than it sounded fucking cool. The thing is, what I had running worked fine and there was no reason to do a
full rewrite besides getting rid of MongoDB.

Over time the codebase turned into legacy code: python 2, running a very old version of mongodb, a very old version of the pymongo driver, a very old version of RQ and little motivation to change things that worked just fine. Every month the mongodb process went down, but restarting it fixed the problem. My solution was a script that restarted the process every day and so, I stopped getting grafana alerts. The API also turned legacy, using a very old version of flask. And still, everything worked fine – I just focused on maintaining pybikes and living my life.

I really do not remember what prompted me to start the rewrite. Existential dread, python 2 EOL, someone complaining that I wasn’t running updates often enough, a new server being unable to run my own software without alternative repos, or this nagging feeling that I had been close to 10 years talking about this multi agent paradigm shit without really trying it. Around June I was doing job interviews and bored of it I started working on the rewrite that became hyper.

Citybikes Hyper is a high-concurrency task scheduling system built for scraping bike-sharing networks using pybikes. Network updates are published over a ZeroMQ PUB socket, allowing external components to subscribe to live data updates.

I hope the guild of architecture astronauts welcomes me

Ok, be wary of buzzwords. It wraps pybikes around a thread pool to use async on it, and start a shit-ton of async workers that consume update tasks from a local queue. Since getting info from the internet is mostly IO bound, async works very well. In the past I was using hard queue systems (celery and then RQ), and always struggled with resources. The system is highly (hyper?) configurable and I probably went overboard with it.

In less than a month I had a full rewrite of my systems already running in production, including the API, for which I just threw away all the old code. No fancy models, no nothing. A subscriber writing to a sqlite db and an api.py file querying the sqlite.

I set it up on a new server, ran it for two days, pointed the DNS record to the new server, killed the old servers and that was it. Nobody noticed, except now my Linode bill had two entries less. But now, oh boy, I had checked another of my conditions for adding historics to citybikes:

A system I could plug in and out of the project without affecting the reliability of the whole system.

This avoids the locking me again in the mongodb scenario: free to try out as many over hyped datastores I feel like without compromising on a particular one. Heck, if I had wanted I could have written a compatibility layer to keep the old systems running as they-were.

I know shitting on micro-services and event driven systems is the new thing, but believe me that for me, on this particular scenario, it is working out fine and using less resources than before, but maybe that is just because I wrote the previous version ages ago.

And with this out of the way, I was finally free to work on adding historical data to Citybikes.

Fear and loathing in bike share

Leave a comment

Storing historics of bike share information has been a running topic for me.

The first time I tried I failed miserably. I was using MongoDB as my primary store for the real time information. I added a new collection called “stats” and nonchalantly started throwing data at it. It was the time of hype and webscale things that together with my own naivete made me think the tool would solve the problem for me. I left it running for some time, letting the db get heavier and heavier, whilst I moved to writing a sample endpoint to query this “stat” information. I excitedly shared this endpoint with some interested people from the internet and left it at that.

Queries ran acceptably for a while until they started crashing my system, which was running on a low spec VM. Fine, I decided to ignore the query endpoint and just focus on archiving the information somehow. The database kept growing until it filled all the available space on the machine. Now what? I downloaded a copy of the 70ish Gb database, rm’d it from the server and started over. The next time that it happened I just disabled the flag for the stats feature and moved on.

No matter what I tried to do with this big fat mongo database, I failed. 70 Gb is far from being considered “big data”, but at the time my laptop was not a stallion, my servers were among the cheapest available, and my internet connection sucked. I know, excuses. The closest I got to extracting an export out of it was by using free GCP credits and firing up a machine with more ram and storage. And then I decided to just forget about it and let the problem linger for years.

This is a post about my takeaways from this first time I tried

No silver bullet

Be wary of magic tools. They can help (or make it worse), but do not magically solve your problems. For MongoDB in particular, I gave up on trying to make something out of it. The query language felt alienating and severely limited the effectiveness of the time I invested and the features I could add.

And let’s be fair. I didn’t fail because I decided to go with the most hyped tool of the time or because I didn’t understand non-relational schemas. I failed because I didn’t have a plan, and because I didn’t understand the problem I was trying to solve. Once things started breaking down the only option available for me was to back off.

I wanted to try again, but my experience on how things had unfolded left me with a sour taste on experimentation. I stubbornly decided to not work on the problem until I had:

  1. A system I could plug in and out of the project without affecting the reliability of the whole system.
  2. A system I could host myself. Sending the problem away to a hosted provider does not effectively solve it for me and I do not want to be at the mercy of the next invoice. Once you are in and things are running, there’s a hard path out of it, too.
  3. A plan for resources. Even if my first experiment was non optimal, it showed that whatever store I used would grow considerably over time. RAM is expensive, Storage is expensive. I needed a plan to say how much it would cost me to run and maintain this for the next N years.
  4. A plan for funding. CityBikes is my side project and it costs both time and money to run. Unless it becomes ‘a project’, it does not make sense to even think about historics, given all the previous points. QED.

And time went by …

… and my project entered maintenance mode the moment I ran out of time and energy. Luckily there were active contributors on pybikes that kept the project updated, and my sole responsibility was to merge code and keep the systems monitored and running. My primary focus was trying to shave hosting costs and automating the tedious parts of running the system – way too much time invested in setting up rate limiting and key authentication to keep the API under fair usage.

From time to time I would get an email asking me if I had historics for bike-share data. Friends asked too. Every time it felt like a punch to my stomach. “IT’S NOT SO EASY OK?”, but I kept feeling I was overcomplicating things, and what’s worst, that everyone thought I was overcomplicating things.

A plan for funding

I started my career as a software developer, leaving the project aside as a background process. Still, most of my roles have been tangentially related to Citybikes. At scrapinghub I was writing spiders, parsers and systems to extract information, much on the same vein of Citybikes. At Kong I was writing lua for the gateway, which was also a tool I was using in Citybikes. When they were called mashape, they had an integration for Citybikes too, so it was all going full circle. Secretly, I thought, I was just learning how to write better software and somehow “getting funded” for keeping the project running.

I have tried at least two times to take a break and focus on Citybikes and it has never really worked out for me. I had the funds, and yet the project was not funded per se, so I would try, and burn out easily.

The last straw was between 2019 and 2021 when I tried to negotiate a contract with one of the companies using the Citybikes API commercially for their product. To me it made sense they would compensate my efforts for running the project for them. After rounds of going back and forth with an SLA that was getting nowhere, I realized I had already spent more on that deal than I would ever take out from it, so I shut down all negotiations and blocked them from using the project. I left from that experience taxed both financially and mentally, and I was very close to just shutting down the project.

To this day, their systems are still trying to hit the API and getting 401d and I am still not very sure what’s going on, really. My only guess is that they have forgotten two rogue machines running forever. I also suspect they have another process running their requests through a proxy service provider, since there is a suspicious number of IPs with the same number of requests and pattern that correlates with their two machines. To me it’s funny to think that they could be running their own instance of Citybikes if they wanted, and decide not to do it.

Exhibit 1: their servers hitting the authenticated endpoint with a key that is no longer valid.
Exhibit 2: A bunch of suspicious IP addresses from all over the world, with the same user agent (very common, but still), each with 9 hits over 15 minutes. Endpoints have 18 requests each, which tells me there are two processes.
Exhibit 3: Rate at which said requests are coming.

Nowadays I try not to think too much about this (I have a button to filter them out from grafana) and instead focus on all the cool things people are building using the Citybikes API. The following are HomeAssistant instances hitting the API. This metric makes me smile, and so I reach for it whenever I need it.

15 minutes of traffic generated by Home Assistant instances

FOSDEM ’24

I visited the NLnet dev room at FOSDEM ’24, talked with them about all of this, sent a proposal the next week and 8 months and a some emails later I had a greenlight for my grant proposal. And guess what, one of the milestones I included on my proposal was Historical Data, under the following description:

Real time information is immediately useful for displaying the location and status of stations. But interesting insights about bike sharing systems are lost if this information is not stored over time. Time series information about bike sharing systems is specially interesting for research purposes and will broaden the impact that Citybikes as a project can do to society by making it freely available for non-commercial purposes.

For me, this meant conquering my fears and trying again where I had failed before. And that’s what I will write about next.

suckless SQLite schema migrations in python

Leave a comment

It happened again. I wrote something that started with a small CREATE TABLE IF NOT EXISTS statement, that kept growing while manually executing DROP statements and changing it as needed. At some point I knew I would need some kind of migration system.

I’ve been there before. I searched “sql migration tool”. The results where the same as ever, a mixed bag of either declarative schema migration tools, alembic and yet more tools that are alpha, beta and carry an ORM by their hand. And that’s fine, I guess.

It happens that having been there before you know that sunk cost applies to the tool you end up going with. And then you go: how hard really is this, why not build my own. But then again, I remember all the features migration tools give me: going back between migrations, cli, etc. But then again, I also remember the troubles glueing together these tools with test suites, how most of the times I end up writing plain SQL, and so on and so forth.

This time I got lucky and found this blog post that gave me a pass for rolling my own, using PRAGMA user_version to keep track of the current schema.

And this time, I decided to go with it. This is my version, and I will go with this until it breaks and stops doing its thing. It is alpha. But at least, it’s my alpha.

First, there’s a folder with migrations. 

$ ls migrations/
0001_init.sql
0002_add_foo.sql

Each migration starts setting PRAGMA user_version to a number.

$ head -n 1 migrations/0001_init.sql
PRAGMA user_version=1;
$ head -n 1 migrations/0002_add_foo.sql
PRAGMA user_version=2;

And then, there’s this I run somewhere.

import os
import sys
import sqlite3
import logging

log = logging.getlogger("foo")

DB_URI = os.getenv("DB_URI", "foo.db")
conn = sqlite3.connect(db_uri)

current_version, = next(conn.cursor().execute('PRAGMA user_version'), (0, ))

migrations = resources.files('migrations').iterdir()

for migration in migrations[current_version:]:
    cur = conn.cursor()
    try:
        log.info("Applying %s", migration.name)
        cur.executescript("begin;" + migration.read_text())
    except Exception as e:
        log.error("Failed migration %s: %s. Bye", migration.name, e)
        cur.execute("rollback")
        sys.exit(1)
    else:
        cur.execute("commit")

And then I am done. Happy that I do not need to spend more time deciding between tools and if ORMs are my thing or not.

Update sqlite3 version on base python docker image

Leave a comment

There’s plenty of reasons to update to the latest versions of sqlite, specifically if you want to use JSON functions and operators. But base python docker images come with sqlite 3.40.0 which is missing a lot of the good stuff (JSONB).

I’ve seen many ways of updating sqlite and I am still not sure I understand how sqlite3 is bundled with python—I assume it gets the whatever version was linked during compilation. One of the solutions is to pip install pysqlite3-binary and honestly that sounds fine and is probably what I would go with if I had found the answer before.

There’s another way, which is to download and install sqlite to whatever version you need. Here are some quick Dockerfile sample steps that worked for me.

FROM python:3.12

# Get sqlite 3.47
ENV LD_LIBRARY_PATH=/usr/local/lib:${LD_LIBRARY_PATH}
RUN mkdir -p /tmp/build/sqlite && \
    curl -L https://www.sqlite.org/2024/sqlite-autoconf-3470000.tar.gz | \
        tar xz -C /tmp/build/sqlite --strip-components=1 && \
    cd /tmp/build/sqlite && \
    ./configure && \
    make && \
    make install && \
    python -c "import sqlite3; assert sqlite3.sqlite_version == '3.47.0'" && \
    cd / && rm -rf /tmp/build

# ...

Exciting things coming for Citybikes

Leave a comment

I am happy to write here that I am starting on a Commons Fund project with NLnet to scale up and develop CityBikes!

While writing this post, I just realized I have barely talked here about the project. So here’s a quick summary on how it started and were we are at the moment.

I started this project back in 2009 when I wanted to build a FOSS android app for the local bike sharing system in my city. I discovered that the information was not publicly available. I first created openBicing, an android app for the local bike sharing system in Barcelona, together with a public API to access the information. Soon I discovered this was not just a local issue, but something that was happening to other cities too, so I created CityBikes with the goal of solving this problem once and for everyone.

What started with an app, made me aware of the importance of open access information compared to visualizations. Data acts as an enabler of creativity, research, transparency and advancement. Representations provide immediate value, but are constrained to a single purpose.

This made me focus all my efforts into the API and pybikes, leaving aside the (popular at the time) android app. Maybe a regrettable decision, since I lost the only revenue channel I had (donations through an app) but also has proven a success in terms of what open data can accomplish. There is an app using the CityBikes API for every platform you can build apps on: not only iOS or Android, but less known platforms too like the pebble, fitbit, telegram bots, Home Assistant, among others.

Over the years, the API has seen considerable increase in usage, and pybikes has attracted a fair number of contributions. All this has been happening organically, and the time I have directed to the project has been very dependent on my free time and excitement and engagement with what I do. As most of everyone that know me know, it’s also been frustrating at times. Spending time monitoring usage of the API to learn that well funded companies are using Citybikes for their projects with zero regards into contributing back either with code or by covering infra costs — I never signed up for that, and it’s not how I like using my free time and my energy tokens.

So what is coming? I like to think that Nlnet is funding me with more time and energy tokens to invest on the project, and hope I will have more things to write here about it. That’s a bit of an empty promise, blogs being blogs, but will see where we go from here.

Onwards! (rip Geoff).

Bash is glue, bash is tape, bash is my friend

Leave a comment

For the last five years I have been using bash more and more. Whilst most ninja-black-ops-rockstars-part-time-pro-surfers moved into writing cool shit in rust or go, I found out most of my day to day problems with technology could be solved with just bash, probably to the detriment of my hireability. Well, I guess it really sucks that my problems are the equivalent of fixing stuff at home and the reason why you do not need a CNC machine for most household mainteinance operations.

My toolbox has a random selection of things I have accumulated over the years; my favourite tool at the moment is a staple gun. “Bash is my staple-gun, bash is my teflon tape and my zip tie” did not have the same ring to it though. Arguably using glue sucks and taping things is not the most elegant way of solving problems, and yet we have all seen cars that are kept together for years with just duc[kt] tape.

What I want to express forward is that it is possible to write useful scripts in bash. If I was a better writer I would try to convince you that these scripts can also be elegant. But I will pass on that for now, since words would fail me. I am no writer but a grunt.

Going back to the original point: why is bash glue, tape, or my friend?

Ken Thompson and Dennis Ritchie, key proponents of the Unix philosophy

I live in this little world of command line tools that I use every day. These tools receive arguments, flags, stdin, do some useful thing and exit with a code (0 meaning all was good, anything else is an error). Most times these also stdout and stderr things too. This is my system, there are other systems, but I like this one. Call it the UNIX way. The beauty of it all was eloquently described in 1999 by Neal Stephenson’s essay In the Beginning… Was the Command Line that can be read online.

Most of the time as a grunt I find that the real work has already been done by people more clever than me. Ah, but the interface. That’s my issue. When a task is simple enough that you do not need anything else, some piping is all you need. But other times you need to sprinkle a bit more magic to it. And here is when bash comes into play.

I use bash to glue together all these clever pieces of code that output exit codes, and as long as these parts do their thing I can move on to doing other stuff. What I find is that bash is unparalleled for that task. And that’s because bash is the shell itself and bash is ubiquitous. With bash I can construct better interfaces to build my own tools.

Sadly I think this puts me way in the middle of the bell curve, but who cares.

The other day I wrote a little snippet to quickly open remote-viewer (SPICE) on VMs that roam on a proxmox instance. The proxmox interface allows you to click through submenus and download a config file to be opened with remote-viewer. That works, but is not the interface I want. Here is my snippet

remote-viewer <(
    curl "https://$HOST:8006/api2/spiceconfig/nodes/$NODE/qemu/$VMID/spiceproxy" \
         -H "Authorization: PVEAPIToken=$TOKEN" \
         -d "proxy=$HOST" \
         -f -s -S -k
)

And I could have left it there. But instead I wrote px. Because I wanted a better way of declaring these variables, and possibly to continue exploring the proxmox api at my will.

This is not the only tool I have written using bash. While working at Kong, together with kidd, we wrote a nice tool called gojira. This time it was not SPICE configs and proxmoxes, but declaring a better interface for docker [compose]. I recommend reading this very useful Shell Field Guide that kidd wrote, based on the experiences we had writing gojira.

  * .    .   *   ___   .    +    .
 .     .   +    /  /  \   .   .
  + .          / /| - - |         *
       *   .   * | - - - |   *   .
   +     .      |---------|   .  +
               _|(O\_ _/O)
             _|/  (__''__)
           _|\/    WVVVVW    ゴジラ!
          \ _\     \MMMM/_
        _|\_\     _ '---; \_
   /\   \ _\/      \_   /   \
  / (    _\/     \   \  |'VVV
 (  '-,._\_.(      'VVV /
  \         /   _) /   _)
   '....--''\__vvv)\__vvv)      ldb

At another time, tired of blindly punching ANSI escape codes by hand I made vtmft and wrote a post about it.

The gist of it was, not only a better interface for a tool but a better interface for ANSI escape codes itself!

Related to this, I golfed a bit over a minimal progress bar solution and made a self documenting bash tool called barsh. And again, wrote a post about it.

Once I was angry at the complex interface of most tools that manage git hook scripts, and wrote hooka. This time I broke my ethos and wrote a full fledged tool instead of relying on the intelligence of others. The fact that I am not using it much tells me it was an error, but I had fun writing it. Look at this crazy linting function I put together.

Not content with my little toys, I again wrote a snippet to center ascii art here. Although this would be one of the examples where you do not need bash but just clever piping.

Finally

Trying to go for my grunt magnus opus, I have written this example scaffolding script for reaching out every other time I want to start a tool.

#!/usr/bin/env bash

# Some logging functions I like to having around

function inf  { >&2 printf "\033[34m[+]\033[0m %b\n" "$@" ; }
function warn { >&2 printf "\033[33m[!]\033[0m %b\n" "$@" ; }
function err  { >&2 printf "\033[31m[!]\033[0m %b\n" "$@" ; }
function err! { err "$@" && exit 1; }

# Tool name, useful for not having to mess with naming early on and support
# consistency with aliases

EXAMPLE=$(basename $0)

# Anything that can be configured on your script should be also configurable
# through environment variables. Sometimes I use a prefix for outside envs
# and proxy them to whatever name inside the script

SOMETHING=${EXAMPLE_SOMETHING:-some-default}
SMTH_ELSE=${EXAMPLE_SMTH_ELSE:-another-default}

# Usage of the script. Single text and no clever tricks here. I have found that
# having some ascii art representing my script makes me happy.

function usage {
  cat << EOF

                              .     '     ,
                                _________
                             _ /_|_____|_\ _
                               '. \   / .'
                                 '.\ /.'
                                   '.'

                $EXAMPLE: an example single-line description

Usage: $EXAMPLE action [options...]

Options:
  -s, --something   set something to anything
  -e, --smth-else   set something else
  -V, --verbose     echo every command that gets executed
  -h, --help        display this help

Commands:
  help                      Show usage

  some-action               Do some action

  another-action <thing>    Another action, takes a thing argument

  yaa <foo> <bar> [...]     Yet another action, takes multiple args

                            Example:
                              $ $EXAMPLE yaa hello world what is up

  ls <foo>                  Foo action, takes an argument. Extra np args get
                            passed to whatever it's doing

                            Example:
                              $ $EXAMPLE ls foobar
                              $ $EXAMPLE ls foobar -- -al

EOF
}


# Parse arguments. I do not use getopts because I can't remember any tar flags
# Important to note that unparsed arguments get stored into ARGS and to stop
# parsing after --  , storing it into _NP_ARGS. Also accept passing arguments
# as stdin with -

function parse_args {
  _ARGS=()
  _NP_ARGS=()

  ! [[ $1 =~ ^- ]] && ACTION=$1 && shift
  while [[ $# -gt 0 ]]; do
    case "$1" in
      -V|--verbose)
        set -x
        ;;
      -h|--help)
        usage
        exit 0
        ;;
      -s|--something)
        SOMETHING=$2
        shift
        ;;
      -e|--smth-else)
        SMTH_ELSE=$2
        shift
        ;;
      -)
        _ARGS+=("$(cat "$2")")
        shift
        ;;
      --)
        shift
        _NP_ARGS+=("$@")
        break
        ;;
      *)
        _ARGS+=("$1")
        ;;
    esac
    shift
  done
}


# Maybe some function that does things
function something-more-complex {
  local foo=$1
  local bar=$2

  set -- "${@:3:$#}"
  inf "Foo: $foo" "Bar: $bar" "Stuff: $*"
}


function main {
    parse_args "$@"

    # re-set action arguments after parsing. Now we can access action arguments
    # in their $1, $2, ... order
    set -- "${_ARGS[@]}"

    warn "SOMETHING=$SOMETHING" "SMTH_ELSE=$SMTH_ELSE"

    case $ACTION in
        some-action)
          echo "Hello World"
          ;;
        another-action)
          [[ -z $1 ]] && err! "Please provide a thing"
          echo "Here is the thing: $1"
          ;;
        yaa)
          something-more-complex "$@"
          ;;
        ls)
          # Some example that uses _NP_ARGS
          inf "The first argument was: $1"
          inf "And this is the result of 'ls ${_NP_ARGS[@]}'"
          ls ${_NP_ARGS[@]}
          ;;
        help)
          usage
          ;;
        *)
          # Display help on unrecognized action but err exit code
          usage
          exit 1
          ;;
    esac
}


main "$@"

Thanks for reading. And thank you bash for being my glue, my tape and my friend.

Note to friends: being glue or tape is not the only thing that’s holding our friendship together. Although, now that I think about it, maybe it is.

You do not need [tool] for simple gitops (most times)

Leave a comment

My starting title was “You do not need ansible for simple gitops”, but writing it I realized I was using ansible as a straw man and in reality what I just wanted to trash about was the general complexity of our professional delivery pipelines.

There’s a big gap between ssh-ing into remote servers and issuing commands by hand (considered bad) and having an [over]engineered pipeline that depends on a variety of random tooling (the eidetic perfection of a platonic ▲).

If your deploys are simple and can be done on an executable script and you are only managing a handful of servers, a bare git repo and some bash is all you will need.

Step 1: create a git user on a server.

You can call this server your git server, or your manager server, ozymandias, or whatever. Set up public and private keys accordingly so you do not need to write passwords every time.

Step 2: create a bare repo on which to push.

You decide how complex you want to make this. Maybe this repo will have its own codebase related to deploying. Or maybe this repo will just be a mirror of another repo that lives in github.

git init --bare mytool.git

Step 3: add an executable post-receive hook.

This script will be called every time you issue a push on this remote.

The following scaffolding script gives you $oldrev $newrev and $refname values that you can use if you want to deploy a particular version. Never trust deploy scripts that uses only master.

#!/usr/bin/env bash
# /path/to/mytool.git/hooks/post-receive

while read oldrev newrev refname
do

  echo "Deploying $newrev"
  /some/path/to/a/deploy/script
  echo "Done"

  # or maybe you need to deploy to other servers

  ssh -T someuser@someserver << EOF
    pushd /some/path/somewhere
      ./deploy $newref
    popd
  EOF

  ssh -T someuser@anotherserver << EOF
    pushd /some/path/somewhere
      ./deploy $newref
    popd
  EOF

done

Step 4: Write a deploy script

How does a simple deploy script look like? In general, it will look very similar to all the steps you manually punch every time you are deploying.

#!/usr/bin/env bash

ref=$1

# stop some services maybe
supervisorctl -c supervisord.conf stop all

# install something
pip install mytool@$ref --upgrade

# start some services again
supervisorctl -c supervisord.conf start all

pushd some/other/path
  ./run-a-bunch
  ./of-more-scripts

  # or maybe you have some docker stuff around
  docker compose pull
  docker compose restart

  # you get my point
popd

Step 5: Add your deploy server as a remote 

git remote add ozymandias git@some-server:mytool.git

Step 6: Deploy any time you want by just pushing to that remote and rejoice.

# upgrade
git push ozymandias master

# downgrade
git push ozymandias master~1

What if?

What if your git server gets compromised and a bad actor issues deploys for you?

Too bad. If your git server gets compromised, you can assume all your systems are compromised too. Try to look at the bright side of it, now you have a bad actor issuing deploys for you, less work!

What if you want an auditable track of your deploys / multiple users with roles and permissions / productivity graphs?

Use something else, like jenkins. Or start investing time in a neverending rabbit hole of external services poorly attached together with duct tape and yaml.

What if my setup is more complex than this, my servers are not pets but cattle and flock, etc?

I would say then you are already knee-deep on the mud and already drowning in complexity. There’s nothing I can say that will help you escape your fate.

Turn all your microservices into little helm charts, use terraform to bring your infrastructure up and down, remote build your docker images, push everything into private registries, write ci actions where every step runs on a docker in docker containerized alpine inside an ubuntu-latest container, do a blue green deploy on your kubernetes clusters, write thousdands of yaml lines and put them into the next shiny tool we have decided will bring us one step closer to

the eidetic perfection of a platonic ▲.

wordpress + sqlite quick start with docker compose

1 Comment

Today I have been trying out the sqlite plugin for wordpress that will reach wp Core on a near future. I did found different instructions out there, but was looking for something I could spin up and down without having to either require an existing running instance, running manual steps or trust non official images, so I ended up writing a docker compose project reference to set the whole thing up. Note the following is for local play testing purposes only.

The repo includes a db.php file (copied from the plugin db.copy file), that gets filled up during the plugin activation. Instead of doing that, it is included and already filled in. WordPress also wants a wp-config.php file present to tell it it has been configured, and the configuration steps require a database connection. I also wanted to skip this step. I also went overboard and ended up running the wordpress alpine with php-fpm image instead of the stock image starting an apache server, so it also includes an fpm.conf file containing a s[ai]mple configuration to nginx alongside.

The following steps will roughly get you there:

$ git clone https://github.com/eskerda/wp-sqlite-docker-compose
$ cd wp-sqlite-docker-compose
$ git clone https://github.com/wordpress/sqlite-database-integration
$ docker compose up -d
$ open http://localhost:8080
$ docker compose logs -f

These are the contents of the docker compose file

version: '3.8'
services:
  wordpress:
    image: wordpress:6.1.1-fpm-alpine
    volumes:
      - wordpress:/var/www/html/
      - ./wp-config.php:/var/www/html/wp-config.php
      - ./db.php:/var/www/html/wp-content/db.php
      - ./sqlite-database-integration:/var/www/html/wp-content/plugins/sqlite-database-integration
      - ./wp-content:/var/www/html/wp-content
  nginx:
    image: nginx:1.7
    volumes:
      - ./fpm.conf:/etc/nginx/conf.d/default.conf
      - wordpress:/var/www/html
    ports:
      - 8080:80
volumes:
  wordpress:

It defines a volume to hold all the wordpress files so it survives restarts, and also includes a local bind mount to wp-content for easy playing with the sqlite db. Ideally I would like to bind mount this file directly, but I was getting permission errors and didn’t really want to deal with that pain at the moment.

$ sqlite3 wp-content/database/wordpress.db
SQLite version 3.37.0 2021-12-09 01:34:53
Enter ".help" for usage hints.
sqlite> .tables
wp_commentmeta         wp_postmeta            wp_termmeta
wp_comments            wp_posts               wp_terms
wp_links               wp_term_relationships  wp_usermeta
wp_options             wp_term_taxonomy       wp_users
sqlite> .schema wp_posts
CREATE TABLE wp_posts (
	ID  integer   NOT NULL  PRIMARY KEY AUTOINCREMENT ,
	post_author  integer   NOT NULL default '0',
	post_date   text NOT NULL default '0000-00-00 00:00:00',
	post_date_gmt   text NOT NULL default '0000-00-00 00:00:00',
	post_content  text NOT NULL,
	post_title  text NOT NULL,
	post_excerpt  text NOT NULL,
	post_status   text NOT NULL default 'publish',
	comment_status   text NOT NULL default 'open',
	ping_status   text NOT NULL default 'open',
	post_password   text NOT NULL default '',
	post_name   text NOT NULL default '',
	to_ping  text NOT NULL,
	pinged  text NOT NULL,
	post_modified   text NOT NULL default '0000-00-00 00:00:00',
	post_modified_gmt   text NOT NULL default '0000-00-00 00:00:00',
	post_content_filtered  text NOT NULL,
	post_parent  integer   NOT NULL default '0',
	guid   text NOT NULL default '',
	menu_order   integer NOT NULL default '0',
	post_type   text NOT NULL default 'post',
	post_mime_type   text NOT NULL default '',
	comment_count  integer NOT NULL default '0'
);
CREATE INDEX post_name ON wp_posts(post_name);
CREATE INDEX type_status_date ON wp_posts(post_type,post_status,post_date,ID);
CREATE INDEX post_parent ON wp_posts(post_parent);
CREATE INDEX post_author ON wp_posts(post_author);
sqlite>

Overall, this looks promising and will probably update this wordpress instance to something similar soon™.

As a final note, I wanted to just provide a quick docker oneliner without much of the fpm bind mount complexity I added along the way. The following just works as much:

docker run --name wp-sqlite -v $(realpath wp-config.php):/var/www/html/wp-config.php -v $(realpath db.php):/var/www/html/wp-content/db.php -v $(realpath sqlite-database-integration):/var/www/html/wp-content/plugins/sqlite-database-integration -p 8080:80 wordpress

Oceanic+ app on the Apple Watch Ultra and the perils of subscription based software

5 Comments

Apple’s newest watch, rated up to 100m comes with a depth sensor (usable up to 40m). This is just a small thing among the many features their Übermensch watch boasts, so its implications might have been eluded. The introduction of the Oceanic+ app has made recent news among both tech and diving circles, yet I feel the discussion is missing out on what to me is a turning point.

Apple is not selling a dive computer, it’s selling a platform to already established dive computer manufacturers, such as Oceanic. The question motivating this post is if the platform is going to be open to other manufacturers and individual developers.

Apple is behind the first general purpose programmable dive computer

I used to think (sweet summer child) that the first programmable dive computer was going to be under a community project. There are some, both software and hardware. The most notable are probably the OSTC dive computers from heinrichs weikamp, and not mentioning it here would be a disservice to them. Still, I would not call the OSTC a general purpose programmable dive computer.

Quick primer on dive computers

I would summarize a dive computer as follows: a device that regularly monitors your time and depth underwater and provide you with safety margins and operations (decompression stops). The reason behind these are lenghty and fall under the realm of the physics of diving, and out of the scope of this post.

All of the following are dive computers, under different tech generations, features and price ranges.

Cressi Leonardo
Suunto d6i
suunto d5
Scubapro SOS Deco meter, aka Bend-O-Matic
shearwater perdix 2 ti, aka you cannot outgrow this computer

All of them do roughly the same. That might come as an overstatement, but on 80% of recreational diving, all of these serve the same purpose: depth, time and a 3 minutes stop reminder. Maybe a compass? For the sake of this post, I will consciously ignore the remining 20%. You know who you are and what tool serves your purpose and your safety. This post is not about you.

Algorithms

A dive computer follows a decompression algorithm. There are many, and are documented mathematic models sprinkled with patented modifications. Bühlmann and RGB are the most common ones. I am mentioning algorithms just to highlight that there’s nothing inherently magical about a dive computer and the algorithm is an implementation detail. The companies behind dive computers are not only selling the watch, but the algorithm.

Why an expensive dive computer

We humans are capricious creatures. We like our tools as much as we like shiny things, and as such, sometimes we buy things that we do not really need for a variety of reasons. It’s none of my business how we humans decide to part with our money, but on the sport of recreational diving, it’s a well known fact that most recreational divers do not need an expensive dive computer. Still, we buy them. Because they are shiny, because they are top of the line. Because they are reliable and have a compass. Because they have a different algorithm or because we cannot outgrow them. And that’s fine! Now again, there’s something very important:

A dive computer that you do not understand is the same as having no computer at all

This much money will give you a very nice dive computer, albeit terribly complex and completely overkill for most of the recreational diving. If someone is willing to spend that much on a dive computer for recreational diving, is an apple watch money better spent?

expensive dive computer that gives you way too much information
a much simpler interface, for the same price

Let’s just acknowledge that a technical dive computer is a tough sell to someone that wants to spend this much but dives very very occasionally, with a dive guide and on shallow waters. I have seen the very confused face of a yacht owner diving with his shiny 1k USD new G2 Scubapro dive computer trying to understand what the numbers are telling him.

For someone that wants to spend this much on a vanity dive computer, a very expensive but simpler dive computer is a better choice. I know that’s what entry level dive computers are for, but think about someone looking for dive computer that is both expensive and simple to use. The market exists.

About target audience

I think the target audience still does not know they are the target audience, because they still do not need a dive computer! For all I know, the target audience still does not even know they want to dive yet.

If you already dive, and you own a dive computer, you are only a very small part of the target audience, and I think it will dwarf compared to the number of people that will get certified already owning an apple watch. Why buy a new dive computer, when the watch you are wearing is already ‘dive computer’ capable?

excuse my crude inkscape art

Subscription: a partnership between Apple and Oceanic

Apple is not a dive computer company. They sell computers. Other dive computer brands also sell computers, but these are not general purpose. Apple is now entering a different new field, it’s offering their platform to others to market their “dive computer experiences”. This is new for scuba diving. I guess it’s not so new on trekking and hiking. A suunto or a garmin might give you a better overall experience, but for most of the recreational hiking, an apple watch loaded with a maps app suffices.

Oh, but the margins… I guess the margins of selling software and hardware are not the same. And as such, Oceanic has decided to go the route of a subscription model. Maybe they are testing the waters? (heh).

By looking at the numbers, tissue loading and decompression info is only covered under a subscription. Even if 80% of recreational diving does not require this, if you are not covering for these, why do wear a dive computer at all? This subscription model makes me sad and I am worried about the implications of this when / if it proves successful.

Remember when I said the physics of diving were out of the scope of this post? Imagine if I told you, pay me 4.99 USD/day and I will tell you about the physics of diving.

The perils of proprietary subscriptions are many, see https://www.gnu.org/proprietary/proprietary-subscriptions.html for more information. A life safety device tied under a subscription that will refuse to provide vital information when the situation needs it raises many questions.

I feel sad knowing that new divers will enter the space with such a model already in place. We tend to forget about the things we have never experienced differently and as such, to someone new a subscription model on a dive watch might not raise the same alarms as it does to me.

It’s this strange gut feeling telling you that something wrong is happening, even though you cannot put it on proper words, and I am trying to do this by writing this post.

What can we do about it (if we care) – mostly questions or a summary

  • Is this a unique deal between Apple and Oceanic, or would they allow other stablished manufacturers to coexist on their platform, say Suunto.
  • Will other manufacturers follow suit, and publish their own diving app for the apple watch. Long term, will they try the subscription model too if they see that it works?
  • Related to the previous point, can the user of an apple watch choose freely which app they want to use for diving?
  • If so, is the community ready for providing a libre diving app for the apple watch? Will the apple store ever accept such an app? Does any apple watch app have unrestricted access to the pressure readings.
  • How long until other platforms start integrating depth sensors and serious depth ratings? The manufacturing technology is there. Once the manufacturing process of integrating these chips exists, any other company will start doing it. If so, will it be an android device?
  • How long until a community platform, such as https://www.pine64.org/pinetime/ starts integrating a depth sensor and is the community ready for such thing happening?
  • I feel that discussing the Oceanic+ on the apple watch hero just from the standpoint of a dive computer is missing the point. If I understand this correctly, Apple is selling their platform to already established brands. It’s telling them, stop building hardware and just build software on mine.
  • I also somehow feel we (as in, me and whoever else cares) have failed our future selfs by being late to this party. We have no platform, no software and the first example of such dive app being subscription based is setting a dangerous precendent to our already walled digital lives.

A docker compose command that brings up a service without a command

Leave a comment

Here is a fun one I got whilst trying to justify why abusing docker compose accepting file inputs from stdin is a very powerful pattern

# Bring up services without a command
#
# usage:
# nocmd.sh up -d service_name

function _compose {
  docker-compose -f docker-compose.yml $@
}

function nocmd {
  cat << EOF
  services:
    $1:
      command: tail -f /dev/null
EOF
}

function nocmd_compose {
  local service=${@: -1}
  _compose -f <(nocmd "$service") $@
}

nocmd_compose $@

Log in and pull from a docker private registry with Rancher Desktop (macOS)

Leave a comment

If you are struggling to get Rancher Desktop to pull docker images from a private container registry, this is what worked on my setup.

It turns out that Rancher Desktop on macOS uses Lima under the hood to start a linux VM and basically, you need to authenticate any docker private registry from within the linux VM.

First, you need to access the Lima VM. Rancher Desktop bundles its own lima binaries and paths, so you need to specify a LIMA_HOME env var. limactl binary is under Rancher’s resources too (or brew install lima and use that).

$ export LIMA_HOME="$HOME/Library/Application Support/rancher-desktop/lima"
$ /Applications/Rancher\ Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl shell 0
lima-rancher-desktop$ uname -a
Linux lima-rancher-desktop 5.15.32-0-virt #1-Alpine SMP Mon, 28 Mar 2022 13:09:00 +0000 aarch64 Linux

From inside the linux VM, it’s now possible to configure docker login credentials. I use a json key file from gcloud https://cloud.google.com/container-registry/docs/advanced-authentication#json-key

lima-rancher-desktop$ cat account.json | docker login -u _json_key --password-stdin https://gcr.io
WARNING! Your password will be stored unencrypted in /home/eskerda.linux/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
lima-rancher-desktop$ nerdctl pull gcr.io/some-gcr-account-1234/private_image
...

If this level of indirection is confusing to you, I want you to know that you are not alone. I did hit this when trying to try out a local helm chart I wrote against something and used rancher as an easy way to get kubectl on macOS. Note to self: there’s no easy way, and it’s not possible to use tools you do not understand. The lack of documentation on this is surprising, and google search sending me here did not help at all. It might just be an impression, but I feel google search keeps getting worse and worse on these kind of queries.

On the positive side, I found out about lima-vm/Lima, which is pretty cool (when forced to use macOS, that is). The following pretty much sums it up

$ uname -a
Darwin trouble 21.5.0 Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:37 PDT 2022; root:xnu-8020.121.3~4/RELEASE_ARM64_T6000 arm64
$ lima uname -a
Linux lima-default 5.15.0-39-generic #42-Ubuntu SMP Thu Jun 9 23:44:48 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

Centering ANSI images on a terminal

Leave a comment

Nothing screams last decade like CSS centering. Nowadays it’s all about the terminal. Say you have this nice snorlax on a text file. It all looks nice, except it’s not centered.

It’s full of ANSI escape codes, so fmt won’t do it

We could clean it up

Get the length of the longest line

Get the width of the terminal, subtract the length and divide it by two

Finally use that magic number and pad your snorlax

Ah yes, a nice oneliner

you know.. let’s make it nice and readable 

function center {
  local f p s
  f=$(mktemp);p=${1:-$(tput cols)};s="s,\x1B\[[0-9;]*[a-zA-Z],,g";cat>$f
  s=$(cat $f|sed $s|while IFS= read -r l;do echo ${#l};done|sort -n|tail -1)
  s=$(((p-s)/2));s=$(printf "%${s}s" " ");cat $f|sed -e 's/^/'"$s"'/';rm $f
}

things I like: vim-fugitive + vim-rhubarb + GBrowse

Leave a comment

There are many things any git plugin for your editor will give you. For me, the most important feature I need to feel productive is any means of opening a particular file in github or any other hosted SCM. I have seen extended git[hub] metadata directly visible at the editor (looking at you vscode) but that feels too much in your face.

Like it or not, your work directory lives on this particular platform, so sooner or later you will have to interact with it. For vim, that’s exactly what :GBrowse is, provided by vim-fugitive and vim-rhubarb. 

:GBrowse                Open the current file, blob, tree, commit, or tag
                        in your browser at the upstream hosting provider.
                        Upstream providers can be added by installing an
                        appropriate Vim plugin.  For example, GitHub can be
                        supported by installing rhubarb.vim, available at
                        <https://github.com/tpope/vim-rhubarb>.

:GBrowse {object}       Like :GBrowse, but for a given |fugitive-object|.

:{range}GBrowse [args]  Appends an anchor to the URL that emphasizes the
                        selected lines. This also forces the URL to include a
                        commit rather than a branch name so it remains valid
                        if the file changes.  You can give a range of "0" to
                        force this behavior without including an anchor.

:GBrowse [...]@{remote} Force using the given remote rather than the remote
                        for the current branch.  The remote is used to
                        determine which upstream repository to link to.

:GBrowse {url}          Open an arbitrary URL in your browser.

:[range]GBrowse! [args] Like :GBrowse, but put the URL on the clipboard rather
                        than opening it.

I have this on my .vimrc. I prefix some macros with , and a theme g[it]. These macros work both on normal, and visual mode for ranges.

map ,gg :GBrowse<cr>
map ,gb :Git blame<cr>

You won’t try it unless there’s some heavy rich and colorful animations flashing on your screen.

Open markdown files instead of messing with local rendering
view a file or a particular set of lines
View a file or a particular set of lines
flow through the history of time and space with git blame
Flow through the history of time and space with :Git blame

things I hate: daylight savings time

Leave a comment

The time was right. Then suddenly it was not. Now it’s always one hour too late.

Not this time. I won’t change the time on my watch and I will try to remember to add one hour to anything that is not looking at the time and saying: this is fine.

We are a total utter failure for allowing our devices to silently change their clocks. I have a crap ton of notifications on my phone from things I couldn’t give less fucks about and yet somehow it manages to not tell me about this. We deserve it.

quick and friendly bash terminal colors

Leave a comment

Just published vtfmt, a small bash utility to quickly colorize strings without either dragging in a huge library with too much features or having to manually put (and remember) ANSI escape codes. I have tried to keep it as minimal as possible and I am still surprised how far can just three lines go!

The implementation is pretty straightforward. An associative array with friendly names for the different ANSI color codes and a function that appends these as a mode between \033[ and m :)

#!/usr/bin/env bash

declare -A FMT_SET=(
  # Set
  [reset]=0
  [bold]=1
  [dim]=2
  [underline]=4
  [blink]=5
  [reverse]=7
  [hidden]=8
  # fg colors
  [fg:default]=39
  [fg:black]=30
  [fg:red]=31
  [fg:green]=32
  [fg:yellow]=33
  [fg:blue]=34
  [fg:magenta]=35
  [fg:cyan]=36
  [fg:light-gray]=37
  [fg:dark-gray]=90
  [fg:light-red]=91
  [fg:light-green]=92
  [fg:light-yellow]=93
  [fg:light-blue]=94
  [fg:light-magenta]=95
  [fg:light-cyan]=96
  [fg:white]=97
  # bg colors
  [bg:default]=49
  [bg:black]=40
  [bg:red]=41
  [bg:green]=42
  [bg:yellow]=43
  [bg:blue]=44
  [bg:magenta]=45
  [bg:cyan]=46
  [bg:light-gray]=47
  [bg:dark-gray]=100
  [bg:light-red]=101
  [bg:light-green]=102
  [bg:light-yellow]=103
  [bg:light-blue]=104
  [bg:light-magenta]=105
  [bg:light-cyan]=106
  [bg:white]=107
)

function vtfmt {
  local out=(); for comp in "$@"; do out+=("${FMT_SET[$comp]}"); done
  IFS=';' ; echo "\033[${out[*]}m"
}

# script is not sourced
if [[ ${#BASH_SOURCE[@]} -lt 2 ]]; then
  # has arguments
  if [[ $# -gt 0 ]]; then
    vtfmt "$@"
  else
    echo -en "$(vtfmt bg:light-magenta fg:black bold) vtfmt $(vtfmt reset) "
    echo -e "This utility is by no means $(vtfmt fg:green)feature $(vtfmt underline)complete.$(vtfmt reset) "
    echo -e "And yet it can do quite some things considering how $(vtfmt bold)small$(vtfmt normal) it is!"
    echo ""
    declare -f vtfmt
    echo ""

    # or basically anywhere to compose different color modes
    WARN_C="$(vtfmt bg:yellow fg:black) WARN $(vtfmt reverse) %s$(vtfmt reset)\n"
    INF_C="$(vtfmt bg:green fg:black) INFO $(vtfmt reverse) %s$(vtfmt reset)\n"
    ERR_C="$(vtfmt bg:red fg:black)  ERR $(vtfmt reverse) %s$(vtfmt reset)\n"

    function inf  { printf "$INF_C" "$*" ; }
    function err  { printf "$ERR_C" "$*" ; }
    function warn { printf "$WARN_C" "$*" ; }

    inf "some info"
    warn "you have been warned"
    err "such an error"
  fi
fi

complex docker-compose templates using bash (or anything you can execute)

Leave a comment

A cool trick I have not seen applied much (maybe it’s a bad idea, and that’s why) is using bash as a templating language for complex compose yml files.

docker-compose is a nifty entry level tool to work with Docker, but it has its own caveats and limitations and is not really under active feature development. By using templates and exploiting file descriptors on the -f flag it’s possible to supercharge and extend docker-compose functionality.

To date, compose files only allow for variable substitution, and only for values. That works for simple straightforward setups, but a more complex environment might need its own runtime logic into it. This becomes useful particularly on development scenarios.

#!/usr/bin/env bash
# some.compose.yml.sh

answer() {
  echo 42
}

cat << EOF
services:
  app:
    image: scratch
    environment:
      ALL_THINGS_ANSWERED: $(answer)
EOF
$ ./some.compose.yml.sh
services:
  app:
    image: scratch
    environment:
      ALL_THINGS_ANSWERED: 42
$ docker compose -f <(./some.compose.yml.sh) -p foo config
services:
  app:
    environment:
      ALL_THINGS_ANSWERED: "42"
    image: scratch
    networks:
      default: null
networks:
  default:
    name: foo_default

This example used bash, but note that any executable producing a valid compose yml output will work!

#!/usr/bin/env python3
# some.py
def answer():
    return 42
print(f"""
services:
  app:
    image: scratch
    environment:
      ALL_THINGS_ANSWERED: {answer()}
""")
$ ./some.py
services:
  app:
    image: scratch
    environment:
      ALL_THINGS_ANSWERED: 42
$ docker compose -f <(./some.py) -p foo config
services:
  app:
    environment:
      ALL_THINGS_ANSWERED: "42"
    image: scratch
    networks:
      default: null
networks:
  default:
    name: foo_default

Hope this silly example makes sense. For a real world case of this pattern being applied, check out https://github.com/Kong/gojira/.

Kudos to Rai and Niji for their part on figuring out this pattern.

See also: A docker compose command that brings up a service without a command

abusing ANSI escape sequences for minimal progress bars

Leave a comment

While diving on ANSI escape sequences, it occurred to me it would be fun to do a progress bar by using only the reverse video escape codes ESC[7m and ESC[27m.

Terminal progress bars as a fun coding exercise. There are countless libraries and helpers out there written in a myriad of languages and techniques. The usual and obvious solution is by using block elements like █ and ░, and more resolution can be achieved by using eighth blocks ▏▎▍…

$ echo 'print("█" * 10 + "░" * 5)' | python
██████████░░░░░

What if instead we use the inverted foreground color to draw the bar? That way it would make it very easy to include text on the progress bar.

$ printf "\033[7mLorem ipsum dolor \033[27msit amet\n"

There are downsides though. Escape codes are just rendered, so progress bars will not carry over uninterpreted plain text. Also, resolution is limited to a single block (compared to drawing eighth blocks):

for i in {0..20}; do
  printf "\033[7m%*s\033[27m%*s▏$i\n" $i "" $((20-i)) ""
done

Here’s a minimal bash implementation. It’s a bit unreadable, but that was the fun of playing golf with it. The amount of features per number of lines is pretty great.

function barsh {
  [[ $# -lt 2 ]] && return 1
  local val=$1; local bas=$2; local txt=$3; local wid=$4;

  [[ -z $wid ]] && { [[ -z $txt ]] && wid=$bas || wid=${#txt} ; }
  [[ -z $txt ]] && txt="$(printf '%*s' "$wid" '')"
  [[ $wid -gt ${#txt} ]] && txt=$txt$(printf '%*s' $((${#txt} - wid)) '')

  local per=$(( (wid * val) / bas ))
  printf "\033[7m%s\033[27m%s" "${txt:0:$per}" "${txt:$per:$((wid-per))}"
}

Does it make sense to publish it? I do not know, but you can do the following:

$ curl https://raw.githubusercontent.com/eskerda/barsh/master/barsh.sh | bash

7 lines of bash, freely available under a MIT license :)

https://github.com/eskerda/barsh

on burnout

Leave a comment

Big companies, and indeed most big institutions are made by a silent majority of the defeated. […] They found joy elsewhere, and found distance between themselves and their work.

source

bash + fzf quickie for an agile zoom experience

Leave a comment

Daily standup, monday 1:1, friday retro, backlog grooming, team meeting. Sound familiar? Go to calendar, find the event, press the zoom link, every day, at least once. Does not sound so agile

Not anymore!

Periodic events usually get the same Zoom room ID every time. Drop in a couple of bash lines, put it somewhere on your path and we get a nifty command to go to zoom rooms without having to fish for links on the screen.

#!/usr/bin/env bash

declare -A rooms

rooms=(
  [standup \(daily\)]="https://zoom.us/j/xxxxxxxxx"
  [1:1 \(mon\)]="https://zoom.us/j/xxxxxxxxx"
  [team meeting \(tue\)]="https://zoom.us/j/xxxxxxxxx"
  [backlock groom \(thu\)]="https://zoom.us/j/xxxxxxxxx"
  [retro \(fri\)]="https://zoom.us/j/xxxxxxxxx"
  [demos]="https://zoom.us/j/xxxxxxxxx"
)

room=$(IFS=$'\n' ; echo -en "${!rooms[*]}" | fzf)
[[ -n $room ]] && open "${rooms[$room]}"

If you ever find yourself on an empty room, go back to step 1, maybe the ID has changed and you are the only one that didn’t get it :)

ゴジラ

Leave a comment 
  * .    .   *   ___   .    +    .
 .     .   +    /  /  \   .   .
  + .          / /| - - |         *
       *   .   * | - - - |   *   .
   +     .      |---------|   .  +
               _|(O\_ _/O)
             _|/  (__''__)
           _|\/    WVVVVW    ゴジラ!
          \ _\     \MMMM/_
        _|\_\     _ '---; \_
   /\   \ _\/      \_   /   \
  / (    _\/     \   \  |'VVV
 (  '-,._\_.(      'VVV /
  \         /   _) /   _)
   '....--''\__vvv)\__vvv)      ldb

Terrible bash is worse than bash

Leave a comment

There’s endless cargo cult about bash being a terrible language and how it should be avoided in favor of go and python for any script longer than N lines. The fact is, no language is going to protect you from being a terrible programmer.

There are many good reasons for using bash where it is suited. There are many good reasons for _not_ using bash where it is not suited.

No matter what the reason is though, there’s no excuse for not using ShellCheck when writing Bash scripts. Every warning and error is clearly documented at https://github.com/koalaman/shellcheck/wiki. It’s an amazing tool, give it a try.

cmdbikes: a terminal client for Citybikes

Leave a comment

Lo and behold cmdbikes, a client to get bike share information right at your terminal.

cmdbikes

You can install it by

$ pip install cmdbikes

Looks good, but why?

Most of the time I spend on Citybikes is either working on pybikes or the API but rarely using it, aka eating my own dog food. For a while I thought the API syntax was sufficiently straightforward to not even need an interface for a language. Turns out I was wrong: the v2 syntax of the API has some caveats and having an abstraction over it helps make everything easier and cleaner.

I first noticed this whilst attending an open data Hackatiño organized by Coruña Dixital and GPUL. Even though I was invited to just be part of the jury, being in a room full of people coding made me want to join in on the fun and prompted me to start working on a telegram bot for Citybikes (which I plan to release on the following weeks/months).

It’s impossible to assume an API is going to have a perfect representation, so abstracting it in some way will always hide the ugly parts and will make building API consumers easier. There are already some interfaces for other languages but I haven’t find any for python, most probably due to the fact that requests is an ubiquitous library that makes things much easier.

Precisely for that reason I decided to create python-citybikes, an interface for the Citybikes API that makes working with it a real pleasure. Some things might change so I will keep it under 0.1 for a while.

The cool part of python-citybikes is that it abstracts all code relating to requesting stuff and from the code point of view, all resources are accessed just when they are needed and saved for later reuse (though an update on the data can be forced).

This snippet, for instance, gives you the first station of the first network on the API.

import citybikes

client = citybikes.Client()
# This will do a request to https://api.citybik.es/v2/networks
a_network = next(iter(client.networks))
# This will do a request to https://api.citybik.es/v2/networks/foobar
an_station = next(iter(a_network.stations))
# This will do no further requests
len(client.networks)
# Neither will this
len(an_station.stations)

This one instantiates a network by its id

import citybikes

client = citybikes.Client()
bicing = client.Network(client, uid='bicing')
# This will do a request to https://api.citybik.es/v2/networks/bicing
print(bicing)
# This will not do further requests
len(bicing.stations)

Another good thing of having an interface for an API is that it allows the addition of utility functions. In this case, I added a near function that will return resources based on distance to a given point.

import citybikes

# Gives you 5 nearest stations on a given lat / lng
lat, lng = 40.7831, -73.9712
client = citybikes.Client()
network = next(iter(client.networks.near(lat, lng)))
stations = list(network.stations.near(lat, lng))[:5]

See where this is going? The best way to assert that an interface for an API is useful is by writing a simple terminal client to play with the API, and that’s what cmdbikes is. Even though it would be fairly easy to bundle cmdbikes together with python-citybikes, I have decided against it. This way I get to feel what’s like to install and use it separately.

This is not the first time I think about writing a terminal client for Citybikes. Some years ago I did a live coding exercise on a talk at pycones 2014 precisely about that. Although, since the talk was about pybikes, the exercise used pybikes and not the API directly.

All in all, writing cmdbikes and python-citybikes has been a useful exercise. For a long time I’ve avoided learning the proper way to publish a package on pypi or writing things on python 3.

Turns out writing a proper package and publishing on pypi is not that difficult (at least anymore) and it’s straightforward to write new python 3 code that is compatible with python 2.7 (at least for a simple case like this).

Revisitando BiciMAD

Leave a comment

Hace dos años publiqué una “auditoria de seguridad” de BiciMAD. Entrecomillado, porque no era tal. Usando un par de herramientas a disposición de la mayoría salieron errores que no se deberían permitir en ningún proyecto público-privado. Éso era sólo la punta del iceberg, se encontraron más gazapos, hubo amenazas, se escribieron artículos y, como es costumbre, se olvidó todo para dar paso al siguiente escándalo.

De no ser por mi proyecto, Citybikes, yo también me habría olvidado. Para contextualizar, Citybikes es un agregador de sistemas de bicicletas públicas y a día de hoy, es la fuente pública más utilizada para crear desde apps a proyectos de investigación que requieran datos sobre bicicletas públicas.

Ante la duda sobre si incluir BiciMAD al proyecto, mi decisión fue la de no hacerlo hasta que no se ofreciera desde el ayuntamiento o la empresa gestora de una fuente abierta y claramente licenciada de datos (para el que no lo sepa, algo conocido como “open data”).

Bien, han pasado dos años. ¿Ha cambiado algo?

La página de BiciMAD sigue mostrando el siguiente ridículo “mapa-imagen”.


Los datos sobre localización y disponibilidad de las bases (que son los que realmente importan) siguen sin ser públicos. Ésto es un extracto de la propuesta en el portal de datos de Madrid.

Información completa BiciMad (NO VIABLE)
Resumen de la propuesta: “Informacion completa BiciMad ”
Fecha de recepción: 30/09/2015
Propuesto por: ciudadano
Estado:NO VIABLE

Actualmente la aplicación biciMad de gestión del servicio público de bicicleta no dispone, en tiempo real, de la información que se solicita, por lo que actualmente no es viable que dicha información se publique en el portal de Datos abiertos por cuestiones técnicas. Creemos que en el plazo de un año, podríamos disponer, con la colaboración y apoyo de la EMT, de una aplicación más enfocada a la actual política de transparencia en beneficio de todos.
Actualmente, ya se publica en el Portal información sobre la bicicleta pública como la posición (georreferenciada) de las estaciones, las incidencias (SYR), media de bicicletas disponibles diaria u usos diarios, tanto de abonados mensuales como ocasionales y el importe pagado en concepto de indicadores de calidad del servicio.

No obstante, si tenemos previsto a lo largo de la primera quincena del mes de junio, publicar en el Portal de Datos abiertos una mayor información sobre biciMad. Entre esta información estarían los datos completos de los gastos del Ayuntamiento por este servicio.

Recientemente ha llegado una contribución a pybikes (una de las partes fundamentales de Citybikes) que añade el sistema de BiciMAD. Dado que, tras dos años, todavía hay excusas “técnicas” sobre la imposibilidad de abrir ésta fuente, no tengo más remedio que aceptar la contribución. Al menos ahora BiciMAD dispone de una API y un mapa online, aunque sea a través nuestro, y sus usuarios podrán elegir entre la multitud de aplicaciones que ya funcionan con Citybikes.

A disfrutar.

Configuring the Polaroid Cube on Linux (or anywhere)

15 Comments

The Polaroid Cube is all fun and games until you see the only way to configure it is through a program for either Windows or OS X.

Fortunately that configuration program does not do anything fancy, it just edits the files Setting.txt and time.txt that live in the root level of the sdcard. These files are read by the firmware of the camera when powered on, and can be edited by any text editor. The syntax is a bit sloppy.

CUBE-V1.01 
UPDATE:N           <--- Set this field to Y before saving!
        FORMAT     <--- No idea
LightFrequency:1   <--- Europe: 50Hz(1), US: 60Hz(0)
TimeStamp:0        <--- Show a timestamp on the topleft corner of videos (dashcam)
CycleRecord:0      <--- Record a video in loop (dashcam)
BuzzerVolume:0     <--- Set this to 0 for disabling the annoying beep. 1 to 50 if you like it
-------------------------------
LightFrequency
	0 ~ 1, def:0, 0:60Hz  1:50Hz
TimeStamp
	0 ~ 1, def:0, 0:Off   1:On
CycleRecord
	0 ~ 1, def:0, 0:Off   1:On
BuzzerVolume
	0 ~ 50, def:5

Setting.txt

UPDATE:N             <--- Set this field to Y before saving!
 
2014-10-20 00:57:44  <--- Set with current date, YYYY-MM-DD HH:MM:SS

time.txt

Notice the two dashcam options. Is it an added feature or a hint of the Cube's hardware origin? Even if it is just a glorified dashcam with a sexy underwater case and the Polaroid brand slapped into it, this toy is well worth the buzz: nice concept, fits on the pocket, has a magnet and feels nice on the hand.

Writing a simple QML client for configuring the Polaroid Cube should be simple enough, but I guess it's not really worth the time: Text is the universal interface, maybe another day.

Auditoria de seguridad: BiciMAD

48 Comments

Tras varios años en CityBikes, cada vez que aparece un sistema nuevo de bicicletas realizo un análisis general sobre el funcionamiento de sus sistemas. Qué sorpresa éste lunes cuando llegué a casa y me enteré de que BiciMAD ya está en marcha.

Ubicación de las estaciones

Normalmente, lo primero que hago es consultar si el sistema pone a la disposición un mapa en su página web: http://www.bicimad.com/mapa.html.

Si miramos el código fuente, podemos ver en la línea #158 como no aparecen las estaciones que deberían pintarse en el mapa. Mi única suposición es que varios servicios ya han estado scrapeando la web y desde el ayuntamiento o bonopark se ha decidido atajar el asunto.

var estaciones = [
					
	                  ];

Pero qué hay de las aplicaciones móviles? De alguna forma accederán a esos datos… El primer paso será descargarse la aplicación de android de bicimad y usarla. Curiosamente, no se puede usar sin estar dado de alta, pero eso tiene fácil solución. Tendremos que decompilar la aplicación y rastrear strings en búsqueda de algún servidor escondido ;) La navaja suiza en éste caso se compone de: Dex2jar, apktool y jd. No es que tengamos especial interés en el código fuente de la aplicación, sólo en las cadenas de texto que representen urls:

grep -r "http://" .
...
./json/JSONParser.java: public static final String URL_SERVIDOR = "http://xxx.xxx.xxx.xx/bicimadMovil/";
./json/JSONParser.java: public static final String url_all_estaciones = "http://xxx.xxx.xxx.xx/bicimadMovil/functions/get_all_estaciones.php";
./json/JSONParser.java: public static final String url_change_password = "http://xxx.xxx.xxx.xx/bicimadMovil/functions/change_password.php";
./json/JSONParser.java: public static final String url_enviar_push_bienvenida = "http://xxx.xxx.xxx.xx/bicimadMovil/functions/send_push_welcome.php";
./json/JSONParser.java: public static final String url_estaciones_cercanas = "http://xxx.xxx.xxx.xx/bicimadMovil/functions/get_estaciones_cercanas.php";
./json/JSONParser.java: public static final String url_generate_password = "http://xxx.xxx.xxx.xx/bicimadMovil/functions/generate_new_password.php";
./json/JSONParser.java: public static final String url_get_datos_usuario = "http://xxx.xxx.xxx.xx/bicimadMovil/functions/get_datos_usuario.php";
./json/JSONParser.java: public static final String url_get_info_tarjeta_consorcio = "http://xxx.xxx.xxx.xx/bicimadMovil/functions/get_info_tarjeta_consorcio.php";
./json/JSONParser.java: public static final String url_get_ranking_usuarios = "http://xxx.xxx.xxx.xx/bicimadMovil/functions/get_ranking_usuarios.php";
./json/JSONParser.java: public static final String url_get_ruta_usuario = "http://xxx.xxx.xxx.xx/bicimadMovil/functions/get_historial_rutas_usuario.php";
./json/JSONParser.java: public static final String url_get_user = "http://xxx.xxx.xxx.xx/bicimadMovil/functions/get_usuario.php";
./json/JSONParser.java: public static final String url_get_user_by_dni = "http://xxx.xxx.xxx.xx/bicimadMovil/functions/get_usuario_por_dni.php";
./json/JSONParser.java: public static final String url_get_weather = "http://api.openweathermap.org/data/2.5/weather?q=Madrid,Spain";
./json/JSONParser.java: public static final String url_registrar_incidencia = "http://xxx.xxx.xxx.xx/bicimadMovil/functions/registrar_incidencia.php";
./json/JSONParser.java: public static final String url_reservar_base = "http://xxx.xxx.xxx.xx/bicimadMovil/functions/set_base_reservada.php";
./json/JSONParser.java: public static final String url_save_new_user = "http://xxx.xxx.xxx.xx/bicimadMovil/functions/set_new_usuario.php";
./json/JSONParser.java: public static final String url_save_ruta = "http://xxx.xxx.xxx.xx/bicimadMovil/functions/set_ruta_usuario.php";
./json/JSONParser.java: public static final String url_submit_form_alta = "http://xxx.xxx.xxx.xx/bicimadMovil/functions/submit_form_tpv.php";
./json/JSONParser.java: public static final String url_submit_form_alta_tutor = "http://xxx.xxx.xxx.xx/bicimadMovil/functions/submit_form_tpv_tutor.php";
./json/JSONParser.java: public static final String url_submit_form_recarga = "http://xxx.xxx.xxx.xx/bicimadMovil/functions/submit_form_recarga_tpv.php";
./json/JSONParser.java: public static final String url_update_user = "http://xxx.xxx.xxx.xx/bicimadMovil/functions/update_usuario.php";
...

Atiza! Bueno, para empezar tenemos ya un enlace a un feed de datos.

curl http://xxx.xxx.xxx.xx/bicimadMovil/functions/get_all_estaciones.php | json_pp
{
   "success" : 1,
   "estaciones" : [
      {
         "porcentaje" : 62.5,
         "latitud" : "40.4168961",
         "longitud" : "-3.7024255",
         "numero_bases" : "24",
         "libres" : "15",
         "luz" : "0",
         "idestacion" : "1a",
         "activo" : "1",
         "numero_estacion" : "1a",
         "nombre" : "Puerta del Sol",
         "direccion" : "Puerta del Sol No 1"
      },
      {
         "porcentaje" : 45.833333333333,
         "latitud" : "40.4170009",
         "longitud" : "-3.7024207",
         "numero_bases" : "24",
         "libres" : "11",
         "luz" : "0",
         "idestacion" : "1b",
         "activo" : "1",
         "numero_estacion" : "1b",
         "nombre" : "Puerta del Sol",
         "direccion" : "Puerta del Sol No 1"
      },
      {
         "porcentaje" : 54.166666666667,
         "latitud" : "40.4205886",
         "longitud" : "-3.7058415",
         "numero_bases" : "24",
         "libres" : "13",
         "luz" : "0",
         "idestacion" : "2",
         "activo" : "1",
         "numero_estacion" : "2",
         "nombre" : "Miguel Moya",
         "direccion" : "Miguel Moya No 1"
      },
      {
         "porcentaje" : 22.222222222222,
         "latitud" : "40.4302937",
         "longitud" : "-3.7069171",
         "numero_bases" : "18",
         "libres" : "4",
         "luz" : "0",
         "idestacion" : "3",
         "activo" : "1",
         "numero_estacion" : "3",
         "nombre" : "Conde Suchil",
         "direccion" : " Plaza Conde Suchil No 2-4"
      },
      ...
   ]
}

Pero aún hay más, sin tan siquiera querer mirar urls tan jugosas como get_usuario_por_dni, el uso de http (no cifrado) para registrarse y hacer login desde la aplicación o analizar a fondo JSONParser nos encontramos con las siguientes joyas:

bicimad
key

Es eso que veo ahí una clave RSA privada para enviar notificaciones push a los dispositivos? Para los legos en materia, suponiendo que la clave esté en uso se podría, por ejemplo, enviar publicidad de parte de BiciMAD a todas las personas con la aplicación instalada. Es decir, suplantar la identidad de BiciMAD. Sin nombrar el resto de scripts de administración que tienen públicos en el listing, PRUEBA.php, etc. Llegados a éste punto decido dejar de hurgar en el bote de gusanos antes de ensuciarme demasiado. Ésto es sólo la punta del iceberg.

Un proyecto enmarcado en el Lote 5 de la contratación de gestión de los servicios públicos de Madrid de 884 millones de euros en los próximos 12 años, con licitación a Bonopark S.L. de 25 millones de euros en 10 años… no lo tengo muy claro.

Me gustaria añadir cómo lo ocurrido aquí defiende la tesis sostenida por CityBikes:

  1. Los datos deben ser abiertos. Más aún cuando el proyecto es público.
  2. Los datos deben ser propiedad del ayuntamiento y nunca de la empresa gestora. Ésta “cláusula” permite al ayuntamiento desarrollar libremente aplicaciones móviles sin depender de la empresa gestora o como me gusta llamarlo, sin que la empresa gestora secuestre los datos.
  3. Los datos son más importantes que las aplicaciones móviles: con datos se pueden fabricar aplicaciones, con las aplicaciones no se fabrica nada (excepto en algunos casos frustraciones).

Recomiendo encarecidamente al ayuntamiento de Madrid seguir los pasos de otros ayuntamientos en política de apertura de datos, como Londres o Barcelona. También al gobierno de España inspirarse en la política de datos abiertos de Francia y su licencia de datos abiertos Open Data Licence que recientemente obligó a JCDecaux a proporcionar una API pública.

En general, la misión de CityBikes pasaría por incorporar éstos datos en nuestras fuentes y ponerlos a la disposición del público de forma libre ya sea para consumirlo desde aplicaciones de terceros como para poder montar proyectos relacionados. Ésta vez, no voy a incluir los datos de BiciMAD en CityBikes ni pybikes hasta que los problemas en sus sistemas informáticos estén “resueltos” y se provea, desde BiciMAD o el ayuntamiento de Madrid, de un feed de estaciones correctamente licenciado, tal como se nos vende desde el gobierno central: http://datos.gob.es/.

Categorizing data feeds at CityBikes

Leave a comment

Heads up for a boring, but needed, post on categorization. CityBikes (API) aggregates to date 176 data sources of different bike sharing systems. One of the main efforts of this project is keeping everything in order to avoid duplicating implementation of feeds that share the same (or at least similar) syntax. In order to understand how to add a new city to the project it’s necessary to know the categorization of feeds inside CityBikes. Let’s make some analysis to be able to name the feeds for what they are.
(more…)

A conky config for thinkpads

2 Comments

Lately I was feeling a bit nostalgic about how the desktop looked when I started playing with linux, and decided that it was time to write myself a Conky config to pimp my laptop.

thinky

Much of the stuff could have been done using the Lua and Cairo API from Conky, but I played it off with just a bash script and iconic fontsets.

thinky

The cool thing about it is that some icons change depending on the state. For instance, the battery icon displays the charge level, and blinks if the charge level < 5%.

Other features include:

  • An indicator for your ThinkLight™
  • Battery status
  • Volume and brightness controls
  • CPU and Memory usage
  • Disk usage
  • Network usage
  • CapsLk indicator

The light indicator, brightness and volume come from the ibm acpi module, thus some changes may be needed to make it work without it.

Installation

git clone https://github.com/eskerda/thinky.git ~/.thinky
cp ~/.thinky/fonts/*.ttf ~/.fonts
conky -c ~/.thinky/conkyrc

Credits

The main font is OswaldFont by Vernon Adams, and the icons provided in the thinky-glyphs.ttf file are part of the following iconic fontsets: Font Awesome, Typicons, Entypo; repackaged into a single file using Fontello.

Dungeon generation using BSP trees

10 Comments

Essentially, a binary space partition algorithm will recursively divide a surface into two for a number of iterations. A binary tree provides the most obvious data structure to traverse the different levels of partitions.

For partitioning, in each iteration the direction of the division -horizontal or vertical- is assigned randomly and so is the size of the first partition, while the second just takes the remaining of the space. Later, on the leafs of the tree (down-most nodes), rooms can be grown within the container, be it randomly or by assigning some constraints that suit the generation. Finally, by connecting the center of each one of the partitions with his brother, all the partitions get connected and accessible.

bsp-dungeon-generation

Maybe it’s a good idea to check the interactive demo of the code that comes ahead before starting. To increase the number of iterations make sure to increase the map size accordingly.

Let’s get into it. First we need a toy implementation of a binary tree. Do not consider the following feature proof or efficient code-wise.

var Tree = function( leaf ) {
    this.leaf = leaf
    this.lchild = undefined
    this.rchild = undefined
}

That’s it. Well, we might sooner or later need some auxiliary functions to get an specific level of the tree and to get the down-most bottom leafs of it, be it for debugging or just for the exercise.

Tree.prototype.getLeafs = function() {
    if (this.lchild === undefined && this.rchild === undefined)
        return [this.leaf]
    else
        return [].concat(this.lchild.getLeafs(), this.rchild.getLeafs())
}

Tree.prototype.getLevel = function(level, queue) {
    if (queue === undefined)
        queue = []
    if (level == 1) {
        queue.push(this)
    } else {
        if (this.lchild !== undefined)
            this.lchild.getLevel(level-1, queue)
        if (this.rchild !== undefined)
            this.rchild.getLevel(level-1, queue)
    }
    return queue
}

Tree.prototype.paint = function(c) {
    this.leaf.paint(c)
    if (this.lchild !== undefined)
        this.lchild.paint(c)
    if (this.rchild !== undefined)
        this.rchild.paint(c)
}

This should be enough to keep things forward. Again, an array mapped tree would avoid us some unnecessary recursive steps, but this is a demo, right?

Now, some utils that will come handy

var Point = function(x, y) {
    this.x = x
    this.y = y
}

function random(min, max) {
    return Math.floor(Math.random() * (max - min + 1) + min)
}

Next thing we need a container prototype. Easy as pie:

var Container = function(x, y, w, h) {
    this.x = x
    this.y = y
    this.w = w
    this.h = h
    this.center = new Point(
        this.x + (this.w/2),
        this.y + (this.h/2)
    )
}

Container.prototype.paint = function(c) {
    c.strokeStyle = "#0F0"
    c.lineWidth   = 2
    c.strokeRect(this.x * SQUARE, this.y * SQUARE,
                 this.w * SQUARE, this.h * SQUARE)
}

Okay, let’s build this tree. We need a function that grows a binary tree of containers.

function split_container(container, iter) {
    var root = new Tree(container)
    if (iter != 0) {
        var sr = random_split(container)
        root.lchild = split_container(sr[0], iter-1)
        root.rchild = split_container(sr[1], iter-1)
    }
    return root
}

And now the actual function that splits a container

function random_split(container) {
    var r1, r2
    if (random(0, 1) == 0) {
        // Vertical
        r1 = new Container(
            container.x, container.y,             // r1.x, r1.y
            random(1, container.w), container.h   // r1.w, r1.h
        )
        r2 = new Container(
            container.x + r1.w, container.y,      // r2.x, r2.y
            container.w - r1.w, container.h       // r2.w, r2.h
        )
    } else {
        // Horizontal
        r1 = new Container(
            container.x, container.y,             // r1.x, r1.y
            container.w, random(1, container.h)   // r1.w, r1.h
        )
        r2 = new Container(
            container.x, container.y + r1.h,      // r2.x, r2.y
            container.w, container.h - r1.h       // r2.w, r2.h
        )
    }
    return [r1, r2]
}

Good, now let’s throw it into a canvas! The wording here might look a bit weird, just assume we are entering this in a development console.

var canvas       = document.getElementById('viewport')
var MAP_SIZE     = 50
var SQUARE       = canvas.width / MAP_SIZE
var N_ITERATIONS = 4

var c_context = canvas.getContext('2d')

var main_container = new Container(0, 0, canvas.width, canvas.height)
var container_tree = split_container(main_container, N_ITERATIONS)

c_context.fillStyle = "#000"
c_context.fillRect(0, 0, canvas.width, canvas.height)
container_tree.paint(c_context)

BSP gone wild
Woosh, not much impressive, this barely resembles a Piet Mondrian scam.

The ugly bits

Randomness it’s all good and fun, until you find out that containers sized too small or too large will result in weird looking results, so here I am discarding any container with a horizontal or vertical ratio smaller than a predefined one. Too aggressive ratios will follow with growing the stack size out of the heap limit.

I do not know if this is the best solution, or if I should not worry about too small partitions, or for instance if I could generate the random width or height with a minimum value (though, I do not like that, as it makes the algorithm code as straightforward. If we define a minimum of 3 squares, what happens when you are confronted by an hypothetical partition of 4 squares?).

function random_split(container) {
    var r1, r2
    if (random(0, 1) == 0) {
        // Vertical
        r1 = new Container(
            container.x, container.y,             // r1.x, r1.y
            random(1, container.w), container.h   // r1.w, r1.h
        )
        r2 = new Container(
            container.x + r1.w, container.y,      // r2.x, r2.y
            container.w - r1.w, container.h       // r2.w, r2.h
        )

        if (DISCARD_BY_RATIO) {
            var r1_w_ratio = r1.w / r1.h
            var r2_w_ratio = r2.w / r2.h
            if (r1_w_ratio < W_RATIO || r2_w_ratio < W_RATIO) {
                return random_split(container)
            }
        }
    } else {
        // Horizontal
        r1 = new Container(
            container.x, container.y,             // r1.x, r1.y
            container.w, random(1, container.h)   // r1.w, r1.h
        )
        r2 = new Container(
            container.x, container.y + r1.h,      // r2.x, r2.y
            container.w, container.h - r1.h       // r2.w, r2.h
        )

        if (DISCARD_BY_RATIO) {
            var r1_h_ratio = r1.h / r1.w
            var r2_h_ratio = r2.h / r2.w
            if (r1_h_ratio < H_RATIO || r2_h_ratio < H_RATIO) {
                return random_split(container)
            }
        }
    }
    return [r1, r2]
}

Ok, run again!

var canvas           = document.getElementById('viewport')
var MAP_SIZE         = 50
var SQUARE           = canvas.width / MAP_SIZE
var N_ITERATIONS     = 4
var DISCARD_BY_RATIO = true
var H_RATIO          = 0.45
var W_RATIO          = 0.45

var c_context = canvas.getContext('2d')

var main_container = new Container(0, 0, canvas.width, canvas.height)
var container_tree = split_container(main_container, N_ITERATIONS)

c_context.fillStyle = "#000"
c_context.fillRect(0, 0, canvas.width, canvas.height)
container_tree.paint(c_context)

BSP with filter ratios
Much more pleasing to the eye. It’s true that a pattern-like structure can be seen and by discarding small results epic randomness is also being discarded, with the possibility to delete rooms that are too small or impossible (hence, resulting in one less room in the map). In any case, the option of discarding is left open, and the results wielded are much more intelligible just for this post.

At the start I was looking after a city-like partition. To see if BSP will yield me what I was looking after, I did set the recursion level to 9 with a map size of 500 squares. The results resemble what I want, and give me enough material to move to the next stage.

Massive BSP

Into the rooms and paths

For the purpose of my generator both rooms and paths are a bit out of the scope as it is not exactly what I am looking for yet. However, I will include them for the sake of completeness, so inaccuracies should be expected.

As mentioned earlier in the post, when we are happy with a recursion level, rooms are placed within each container. The sizing on the room can be defined by any rules. In this example, rooms are grown with a random padding ranging from 0 to a third of the room size (for each of the sides). By allowing it to be 0, the possibility of touching rooms is contemplated, which should give interesting results.

var Room = function( container ) {
    this.x = container.x + random(0, Math.floor(container.w/3))
    this.y = container.y + random(0, Math.floor(container.h/3))
    this.w = container.w - (this.x - container.x)
    this.h = container.h - (this.y - container.y)
    this.w -= random(0, this.w/3)
    this.h -= random(0, this.w/3)
    return this
}
Room.prototype.paint = function(c) {
    c.fillStyle = "#888"
    c.fillRect(this.x * SQUARE, this.y * SQUARE,
               this.w * SQUARE, this.h * SQUARE)
}

Using the helper function to get the leafs of the tree rooms are created dependent of each one of them and later its paint function is called.

var canvas           = document.getElementById('viewport')
var MAP_SIZE         = 50
var SQUARE           = canvas.width / MAP_SIZE
var N_ITERATIONS     = 4
var DISCARD_BY_RATIO = true
var H_RATIO          = 0.45
var W_RATIO          = 0.45

var c_context = canvas.getContext('2d')

var main_container = new Container(0, 0, canvas.width, canvas.height)
var container_tree = split_container(main_container, N_ITERATIONS)

c_context.fillStyle = "#000"
c_context.fillRect(0, 0, canvas.width, canvas.height)
container_tree.paint(c_context)
var leafs = container_tree.getLeafs()
for (var i = 0; i < leafs.length; i++) {
    new Room(leafs[i]).paint(c_context)
}

BSP with rooms

Sweet. Now let’s add a way for paths to be drawn on the canvas. Again, the purpose of this demo is just on drawing the stuff, so we are not going to save them in any structure, nor carve it into a tile map. Let’s see what happens if we draw a line between the centers of containers with the same parent.

Container.prototype.drawPath = function(ctx, container) {
    ctx.beginPath()
    ctx.lineWidth = SQUARE
    c.strokeStyle = "#888"
    c.moveTo(this.center.x * SQUARE, this.center.y * SQUARE)
    c.lineTo(container.center.x * SQUARE, container.center.y * SQUARE)
    c.stroke()
}
var draw_paths = function(ctx, tree) {
    if (tree.lchild == undefined || tree.rchild == undefined)
        return
    tree.lchild.leaf.drawPath(ctx, tree.rchild.leaf)
    draw_paths(ctx, tree.lchild)
    draw_paths(ctx, tree.rchild)
}

var canvas           = document.getElementById('viewport')
var MAP_SIZE         = 50
var SQUARE           = canvas.width / MAP_SIZE
var N_ITERATIONS     = 4
var DISCARD_BY_RATIO = true
var H_RATIO          = 0.45
var W_RATIO          = 0.45

var c_context = canvas.getContext('2d')

var main_container = new Container(0, 0, canvas.width, canvas.height)
var container_tree = split_container(main_container, N_ITERATIONS)

c_context.fillStyle = "#000"
c_context.fillRect(0, 0, canvas.width, canvas.height)
container_tree.paint(c_context)
draw_paths(c_context, container_tree)
var leafs = container_tree.getLeafs()
for (var i = 0; i < leafs.length; i++) {
    new Room(leafs[i]).paint(c_context)
}

BSP, rooms and paths

Good as done. Now, what about some cleaning up?

var Map = function(canvas) {
    this.width  = canvas.width
    this.height = canvas.height
    this.ctx    = canvas.getContext('2d')
    this.c_tree = undefined
    this.rooms  = []
}

Map.prototype.init = function() {
    var m_container = new Container(0, 0, MAP_SIZE, MAP_SIZE)
    this.c_tree = split_room(m_container, N_ITERATIONS)
    this.growRooms()
}

Map.prototype.growRooms = function() {
    var leafs = this.c_tree.getLeafs()
    for (var i = 0; i < leafs.length; i++)
        this.rooms.push(new Room(leafs[i]))
}

Map.prototype.clear = function() {
    this.ctx.fillStyle = "#000"
    this.ctx.fillRect(0, 0, this.width, this.height)
}

Map.prototype.drawGrid = function() {
    this.ctx.beginPath()
    this.ctx.strokeStyle = "rgba(255,255,255,0.4)"
    this.ctx.lineWidth = 0.5
    for (var i = 0; i < MAP_SIZE; i++) {
        this.ctx.moveTo(i * SQUARE, 0)
        this.ctx.lineTo(i * SQUARE, MAP_SIZE * SQUARE)
        this.ctx.moveTo(0, i * SQUARE)
        this.ctx.lineTo(MAP_SIZE * SQUARE, i * SQUARE)
    }
    this.ctx.stroke()
}

Map.prototype.drawContainers = function() {
    this.c_tree.paint(this.ctx)
}

Map.prototype.drawRooms = function() {
    for (var i = 0; i < this.rooms.length; i++)
        this.rooms[i].paint(this.ctx)
}

Map.prototype.drawPaths = function(tree) {
   if (tree.lchild == undefined || tree.rchild == undefined)
        return
    tree.lchild.leaf.drawPath(this.ctx, tree.rchild.leaf)
    this.draw_paths(tree.lchild)
    this.draw_paths(tree.rchild)
}

Map.prototype.paint = function() {
    this.clear()
    this.drawGrid()
    this.drawContainers()
    this.drawRooms()
    this.drawPaths()
}

Which now can be run as

var map = new Map(document.getElementById('viewport'))
map.init()
map.paint()

bsp-dungeon-generation-random

Feel free to check the playground demo at /demos/dungeon.

EDIT: Some years after, I rewrote most of the code from this post, maybe it’s useful /demos/pizza.

The mule at the Blue Systems sprint

Leave a comment

A Mule

At the last Blue Systems sprint in Pineda I found a mule roaring outside the house. I love how friendly these animals are. In particular, this one followed me through the fence while walking up the hill, and when I was out of my bike to snap a picture, decided to take his head off the fence. I can only assume this is his preferred spot to creep and get food from strangers.

Fun with the Arduino UNO and a NES gamepad

5 Comments

A while ago I wanted to play the original NES. It turned out my console was broken and so, decided to find a way to connect the NES gamepad to the computer using an Arduino.

I looked on the Internet and found many things, but not exactly what I wanted:

What I found

  • Read the NES gamepad from an Arduino
  • Emulate a keyboard on the computer, and interpret Arduino serial input as keyboard presses.
  • How to build USB firmwares for AVR microcontrollers. How to build an USB firmware for AVR that acts as a gamepad.

What I wanted

  • A NES gamepad that showed on the computer as an USB game HID: something you can plug into any computer / console and will (hopefully) work out of the box, plug and play.

So, all the pieces for the puzzle were there, and the only thing left was to put them together. Now that it’s working as I intended to I have decided to write this post to save some time to anyone interested in building similar things. This post might be too verbose for the purpose of this thing, but what’s the fun on putting something together you can barely understand? I decided to spend more time than usual inspecting all the things involved on the process.

Note that these are not the instructions on how to do it, but a mix up of story, missing gaps of information I found, or at least some useful information to lower the entrance barrier a bit, and some reference links. For instructions check the repository.

TL;DR: I put together a USB HID gamepad with the NES controller and an Arduino.  Also, first post.

P1011229_t

Onto the NES Gamepad

The NES gamepad has a 8-bit shift register that, upon receiving a LATCH saves the state of the 8 buttons (GND means pressed). Clocking a signal through the CLK line, the state byte is shifted through the DATA pin.

    NES gamepad controller pinout

        o 4    1. +5VDC Supply                            INPUT
    1 o o 5    2. ?
    2 o o 6    3. ?                                 _|_|_|_|_|_|_|_|_
    3 o o 7    4. GND                   LATCH  ____|                 |
               5. CLK                              |     IC 4021     |
               6. LATCH                   CLK  ____|\                |----- DATA
               7. DATA                             |/                |
                                                   |_________________|
           ________
    LATCH |        |
    ______|        |______________________________________________________________________________

                     ____      ____      ____      ____      ____      ____      ____      ____ 
    CLK             |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |
    ________________|    |____|    |____|    |____|    |____|    |____|    |____|    |____|    |__

         _  ________  ________  ________  ________  ________  ________  ________  ________  ______
    DATA  \/    A   \/    B   \/ Select \/  Start \/   Up   \/  Down  \/  Left  \/  Right \/
         _/\________/\________/\________/\________/\________/\________/\________/\________/\______
                     |--12µs--|

A diagram on extracting the input from a NES gamepad. Some pins on the IC 4021 omitted for clarity.

The following code outputs the input from a NES gamepad in binary words.

int CLK = 2;
int LATCH = 3;
int DATA = 4;

byte last_read = 0xFF;

void setup();
void loop();

void setup()
{
    Serial.begin(115200);
    pinMode(CLK, OUTPUT);
    pinMode(LATCH, OUTPUT);
    pinMode(DATA, INPUT);
}

void loop()
{
    byte reading = read_NESpad();
    if (reading != last_read){
         Serial.println(reading, BIN);
    }
    last_read = reading;
}

byte read_NESpad() {
      /*
        NES Word Mapping
        x x x x x x x x
        | | | | | | | |_  A
        | | | | | | |___  B
        | | | | | |_____  SELECT
        | | | | |_______  START
        | | | |_________  UP
        | | |___________  DOWN
        | |_____________  LEFT
        |_______________  RIGHT
     */

  // Send a HIGH pulse to latch. Make 8 shift register store state
  // of all buttons
  digitalWrite(LATCH, HIGH);
  delayMicroseconds(12);
  digitalWrite(LATCH, LOW);

  // Clock the 8 shift register to get the
  // state of the buttons
  byte output = 0x00;
  for (int i = 0; i < 8; i++){
      output |= digitalRead(DATA) << i;
      digitalWrite(CLK, HIGH);
      delayMicroseconds(6);
      digitalWrite(CLK, LOW);
      delayMicroseconds(6);
  }
  return output;
}

This was enough to build a minimum viable hack, consisting on an arduino sketch reading the gamepad and sending it through the serial interface and a Node.js (\o/) server reading it and firing xdotool to emulate key presses. Ugly, but enough for a hit of nostalgia for the rest of the weekend (if you really want to see, it lives on this branch). With this I was on the same as most of the links I found googling around.

After playing with it for a while I sort of felt this was well behind my initial goal. Which I initially wanted was to actually build an USB gamepad you could plug onto your friends’ computers, no install, no anything. Upon researching, I found out that meant flashing a new firmware to the UNO, even if I did not really understand what that meant.

A little guide on Arduino and DFU

Instead of using an specific purpose USB-to-serial chip, the Arduino comes with an ATmega programmed to act as an USB to serial device (so you can upload compiled bytecode from sketches into the EEPROM). That’s what gets detected when you connect the Arduino into the computer. In better words:

The Uno differs from all preceding boards in that it does not use the FTDI USB-to-serial driver chip. Instead, it features the Atmega16U2 (Atmega8U2 up to version R2) programmed as a USB-to-serial converter.
http://arduino.cc/en/Main/arduinoBoardUno

    ____________________
   | o       ···········
RESET ---> + · ·
GND -----> + · ·
  _|____
 |      |
 | USB  |
 |______|
   |
   |

In order for the Arduino UNO to appear (and act) as a different device, a new firmware has to be flashed into the ATmega16U2, by putting it into DFU (direct firmware upgrade) mode which, depending on the board, is done with more or less messy ways which for older versions may require soldering.

Hopefully my board is an R3, which can be cleanly put into DFU mode by jumping the RESET and GND pins on the Atmega16U2 together for a fraction, using a jumper or, if feeling wild, just a piece of copper.

Looking for inspiration

At this point, I did not really know what to do. I could read the NES gamepad from the Arduino using sketches, and was able to flash new firmwares onto the arduino, next step?

LUFA

LUFA (Lightweight USB Framework for AVRs, formerly known as MyUSB) is an open-source complete USB stack for the USB-enabled Atmel AVR8 and (some of the) AVR32 microcontroller series, released under the permissive MIT License (see documentation or project source for full license details). The complete line of Atmel USB AVRs and USB AVR boards are supported by the library, as are any custom user boards, via custom board hardware drivers supplied by the user.
http://www.fourwalledcubicle.com/LUFA.php

Darran Hunt’s Arduino Hacking blog has a nice set of posts with examples on how to build different HID devices using LUFA and Arduino. In particular, there’s one example that emulates an USB joystick with two axis and two buttons, which is almost what I needed. The example is, at the same time, based on one example from the LUFA library, and Arduino’s USB-to-Serial firmware.

HID Devices interact with the USB stack using HID reports. The firmware describes itself as an USB HID Joystick and reads on the serial interface from the main processor (the ATmega328), which means it can read data from the main code (sketch, running on the ATmega328) to the firmware code (running on the ATmega16U2). The sketch handles all the input, and sends it through the serial interface using a packed struct. The firmware just roams there, waiting for data, and then sends it to the USB stack as a proper HID report.

I found Darran’s design on this issue pretty clever and on the line of what Arduino is supposed to feel like. The firmware is doing all the heavy lifting, and all the logic lives on the sketch.

Note that there was no real need to compile a modified version of the firmware. I could be ok by using one example that has 40 buttons, and throttles and more axis. But when I get into something, I really want to understand how all the parts work together. If not, it’s almost time wasted.

HID report descriptors 101

For the USB stack to know what device is it talking to, and how this reports do look like, it uses a descriptor table. In this descriptor table you fill up everything about your device: what’s its purpose, what type of device is it, how many inputs it has and how are they represented. Oh, and it has to be 8 byte aligned, which means adding a padding every time you are sending less than a byte. Everything is documented on the USB HID Specification document (have fun with it).

Some reference links: [wikipedia] [USnooBie’s USB HID Report Descriptor Tutorial 1]

2 Axis
Min Max Value (-100, 100)
2^3 = 8 (size, signed integer), #2 (axis)
4 Buttons
size 1 (boolean), # 4
+ 4bits of padding.

Device and manufacturer ID

Devices identify themselves using a device and manufacturer ID. For acquiring one yourself you would have to pay large sums of money to the USB foundation. I have decided to leave the original Atmel Corp. LUFA Joystick Demo Application.

The whole thing

P1011227_t

Hacking a firmware yourself

You’ll need to:

  • Set up the LUFA environment
  • Read these posts in Darran’s blog, just for reference.
  • Take an example from the LUFA library that suits what you want.
  • Look into arduino’s usb-serial original firmware.
  • Combine them both to interface with the Arduino.
  • Write your HID Descriptor
  • Start hacking.

Chicken and egg

At this point, I assume I could be doing all the ‘nintendo’ parsing part on the firmware, be it by directly interfacing with the INPUT ports or just getting the 8 bits through the serial interface instead of the whole HID Report, but that would make it too specific. Having just a firmware that does one thing, and does it well just keeps everything simpler.

But being that Arduino is meant for easy hacking into microcontrollers, that would not make much sense, and I would be better by just glueing an atmel microcontroller into a NES Gamepad.

Conclusions

After this I would love to put together a project to ease the creation of USB game devices, just by using the arduino library, similar to unojoy. While describing a 30 button – 6 axis – 1 hat switch device could be enough to have a generic firmware, I want my devices to appear on the computer exactly as they are. What I think of would be some compilation variables or a descriptor generator, and then a different protocol of communication between the sketch and the firmware.

Also, I cannot wait to get more gamepads of consoles I did not have, like the SNES and the N64.

Finally, it will be interesting to put two gamepads together on the same board and make them appear as two devices by using report ids on the HID descriptor.