Docker notes

A couple of notes based on my recent efforts to dockerize TV Graph.

1. Environments, etc. created by Docker build stages are only persistant within that stage. That means that something like
...
RUN . ./venv/bin/activate
RUN pip install -U -r conf/requirements.txt
...
won't work because the virtual environment is activated in a different shell than pip is run.
For the same reason you can't do
...
RUN mongod &
RUN mongo < conf/mongo.js
...
as the MongoDB server won't be running in the same environment as the client.
This, however, will work:
RUN mongod && mongo < conf/mongo.js

2. Speaking of MongoDB, the official Docker image for MongoDB marks Mongo's default /data/db/ folder as a Docker volume, which means any changes done to it during the build stage won't be visible after build, which is an issue if you need to change the database during build for example to create indexes (see above).
One workaround is to force MongoDB use a different folder for data storage by using the --dbpath parameter for mongodb. The downside of it is that of course unless you manually map this folder out of the container, it's contents are going to be lost once the container is deleted.

3. For some odd reason, there appears to be some odd sort of a race condition or something when trying to run a MongoDB script during Docker build. The script I was trying to run was very simple:
db = db.getSiblingDB("tvgraph");
db.createUser({user: "tvgraph", pwd: "tvgraph", roles: ["readWrite"]});
db.tvseries.createIndex({show_title: "text"});
db.tvseries.createIndex({"created_at": 1}, {expireAfterSeconds: 43200});
Yet the last line (ini this case, the TTL index) was never executed. Shifting the lines around, adding more lines, adding blank lines at the end of the file didn't help. What worked is inserting sleep(1000) after every statement to give it time to run. Somehow this worked and unfortunately I don't have time to investigate any further.

Hiding unauthorized Samba shares

Another thing that took a while to research but turned out to be fairly trivial.
I was looking for a way to make only those Samba shares appear for a user that they are authorized to access.
The solution is to create a per-user share definitions with those shares that that user can access, then include them in the main smb.conf file like so:
include = /etc/samba/%U.conf
where %U resolves to the name of the user's home directory (other Samba variables won't work reliably for reasons I forget) which almost always matches the user's Samba username.
All the public shares available to all can still be included in the main smb.conf file alongside this definition. Of course, another way is to store users' Samba configuration files in their home directories, which is probably somewhat cleaner, then the include will look something like this:
include = /home/%U/.samba.conf

Samba over OpenVPN

Another thing that I need to write down before I forget.

I wanted to set up access to my home server via Samba that would work both at home through local network that the server is on and remotely, through OpenVPN connection to the server.

While the OpenVPN part is fairly easy to set up, the transparent file access unfortunately is not. I wanted to use a TUN interface for the VPN to lower the overhead on the already slow connection, so the traditional service discovery methods like Avahi did not work. I also didn't want to set up a dedicated DNS server, as it seemed an overkill for this task.

The solution I came to was quite simple and effective. Since all I needed was connection to a single server, I just decided to dynamically change the /etc/hosts file upon connection.
Fortunately, OpenVPN clients for all operating systems are able to run custom scripts on connect / disconnect. For the following, nas is my server hostname and *** is the name of the OpenVPN configuration file.

When this is done, the same nas hostname will be accessible using service discovery while on local network and by using the hosts file record while connecting remotely. Since the hostname remains the same, all references such as network drives and Quick Access folders will still work.

MacOS with Tunnelblick

1. Create connected.sh with the following content:

#!/bin/sh
echo "10.8.0.1 nas" >> /etc/hosts

2. Create post-disconnect.sh with the following content:

#!/bin/sh
sed -i ""  "/10.8.0.1/d" /etc/hosts

3. Put both scripts in ~/Library/Application Support/Tunnelblick/Configurations/***.tblk/Contents/Resources/

Linux

Use the same scripts as above, but include them in the openvpn configuration by setting --up and --down options respectively.

Windows with the official OpenVPN client

1. Create ***_up.bat with the following content:

echo 10.8.0.1 nas >> %windir%\System32\drivers\etc\hosts
exit /b 0

2. Create ***_down.bat with the following content:

cd %windir%\System32\drivers\etc\hosts
findstr /v "nas" hosts > hosts2
del hosts
move hosts2 hosts
exit /b 0

3. Put both in the same folder as the ***.ovpn configuration file.

4. Make sure the OpenVPN GUI is launched as administrator, otherwise the scripts won't get executed with proper permissions and will fail to run.

Transmission reverse proxy

All right, I have to write this down before I forget. I spent the better part of the day figuring this out, so here it is in case I ever need this again.

I wanted to set up a short URL for transmission-daemon on my NAS so that instead of awkward http://nas:9091/ I could type something more finger-friendly (most notably, without the colon for port number), ideally http://nas/t/. Here is the final config that resulted after several hours of research and apache2ctl restarts:

<VirtualHost *:80>
        ServerName nas
        RewriteEngine on
        RewriteRule /t[/]?$ /transmission/web/ [R=permanent]
        ProxyPass /transmission http://127.0.0.1:9091/transmission
        ProxyPassReverse /transmission http://127.0.0.1:9091/transmission
</VirtualHost>

Thanks to everyone on forums and StackOverflow for numerous solutions that allowed me to piece this all together.

Nexus 4 SIM card problems

A while ago, after I got my shiny new Nexus 4, I've suddenly started having a problem with one of my SIM cards - the phone started to randomly lose the connection to the SIM card, and even a restart would not help. The internet was silent - apparently no one had problems that were anything like this.

After three or four hours of experiment and trial, on the verge of sending the phone back to Google to be replaced, I finally understood what the problem was.

Apparently, SIM cards can have contact pads that differ in size and their position on top of the card. It doesn't really matter since the contacts are large enough to allow for such inconsistencies. However, when the contact pad is too close to one edge of the card, it gets short-circuited by the metal SIM card holder of the Nexus 4 as can be seen here:

Once I found this out, the solution was simple: you can put a piece of non-conductive tape either on the card holder or on the SIM card itself to protect the contacts. I chose the latter, as only very few cards have this problem and it is easier to "patch" them. This solution is not exactly glamorous, but quite effective - I haven't had a problem since.

TfL Knows about You

This morning I received a curious example of data mining performed by Transport for London - an email warning that part of the station I use every day will be closed for renovation.

I highly doubt that they sent out this email to all the Londoners who use other stations - how many of them would care about a staircase at Finsbury Park? The only way for them to target me with this information is to data mine my Oyster travels and see which stations I use the most. While it is quite and obvious and expected that they will use the data in such a way, it still a bit scary. Even more so knowing that my Oyster card is registered with my real name and address.

There is nothing like starting your day with a small reminder that someone out there knows a lot more about you that you think.

Skydrive WebDAV Access - Updated

Somehow there is very little information about WebDAV access to Skydrive, which is upsetting, because with 7GB (or 25GB, if you're lucky) of storage it becomes a very nice place to store your files.

Now, Microsoft officially claims Skydrive does not support WebDAV at all, which is sort of funny when you consider that it is exactly what the new Office 2013 uses to allow you to save your files there. Basically, when you save a document to Skydrive from one of the Office apps, it mounts your Skydrive account to a folder, saves a document there and then dismounts it.

Doing this yourself is very simple, and while some older tools (like Skydrive Simple Viewer or Skydrive plugin for Total Commander) don't seem to work, Office is going to help us.

First, connect enter your Skydrive account details in Office 2013 to enable all related capabilities.

Second, open any document, click Save As, select your Skydrive and click Browse. The following window should come up, showing the contents of your Skydrive.

All you need to do now is to click on the address bar and copy all its contents - this is the WebDAV address of your Skydrive account.

Now you can use this address in any WebDAV enabled application in any operating system. The simplest way to see if it works is to map a network disk in Windows. Use your Microsoft account email as username and your password when prompted.

Overall, WebDAV access might be a bit slow at times, but I find having direct access to my files very convenient for certain applications where Dropbox-like syncing might be an overkill.


UPDATE: Having to use Microsoft Office just to get your WebDAV address is highly inconvenient - for Linux users it means they have to have access to a Windows box. Fortunately, as I have just found out, there is a very simple way to get your Skydrive WebDAV address from any OS.

Just log in to your Skydrive web interface at skydrive.com, click your name in the top right corner and select "Edit Profile". When the page opens you should have a similar address in your address bar:

https://profile.live.com/P.mvc#!/cid-5190000000013e3c/

Now take the part after "cid-" (in this case "5190000000013e3c") and add it to the following address: "https://d.docs.live.net/". In our example it becomes:

https://d.docs.live.net/5190000000013e3c

And voila - here is your Skydrive WebDAV address. Again, you can try it out by mapping it as a network drive in Windows Explorer.


UPDATE2: Unfortunately, while the above solution works well in Windows and allows you to access your Skydrive via a mapped network drive, it doesn't seem to be possible to do the same in Linux.

Despite many efforts to mount Skydrive with davfs2 in Linux, I kept getting the "Found 302" error, meaning that the provided WebDAV URL redirects to another page. It turns out, davfs2 currently is unable to handle any redirects, which Skydrive (unlike many other WebDAV storage providers) uses. There is a related davfs2 bug that was filed over 2 years ago, yet it seems no progress has been made since.

Effectively, until this is fixed, there seems to be no way to access Skydrive via WebDAV in Linux.


UPDATE3: To be completely fair, it turns out it is possible to access specific folders of your Skydrive via WebDAV but only when using KDE's Dolphin file manager. This is made possible thanks to the fact that Dolphin uses Konqueror's (Rekonq's) cookies to establish a session, thus if you have previously logged to skydrive.com with Rekonq, it will use the same session and allow you to see your files.

To do that you have to log in to the Skydrive web-interface with Rekonq, open a folder you want to work with and note it's number in the browser's address bar. The number will be the digits after the "!" in the address. Now, insert the following addres in Dolphin's address bar and you should instantly see your folder and work with it normally:

webdavs://cid-5190000000013e3c.users.storage.live.com/items/5190000000013e3c!12345/

In this example "5190000000013e3c" is your ID we've found in the steps above and "12345" is the folder number.

Unfortunately, this only works in Dolphin and you can only work with one folder at a time.